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I.  INTRODUCTION 


A.  MOTIVATION 

Homeland  security  and  military  operations  depend  heavily  on  timely,  accurate 
recognition  of  friendly  and  hostile  assets,  or  contacts  of  interest  (COIs),  to  increase  the 
level  of  situational  awareness  (SA)  and  facilitate  decision-making.  Generally, 
surveillance,  battlefield  monitoring,  and  target  recognition  are  non-stop  processes  that 
involve  several  resources.  These  processes  are  a  major  concern  throughout  all  kinds  of 
operations  and  require  the  engagement  of  various  types  of  sensors,  such  as  radar,  infrared 
(IR),  and  thermal.  The  combined  use  of  these  sensors  is  the  key  for  accomplishing  this 
highly  important  process.  The  platforms  participating  in  such  operations  often  have 
limited  capabilities  to  perfonn  stand-alone  recognition  of  targets  under  any  given 
circumstances.  Therefore,  collaboration  between  platforms  is  necessary.  Fusion  of 
various  data  collected  from  different  platforms  and  a  variety  of  dissimilar  sensors  enables 
the  real  or  near  real-time  recognition  of  COIs. 

Furthermore,  the  ubiquitous  mobility  of  these  platforms  raises  the  challenge  of 
uninterrupted  and  robust  communications  to  facilitate  data  exchange.  The  need  for 
continuous  monitoring  of  available  assets  and  exploitation  of  their  capabilities  in  real 
time  is  more  than  obvious.  Allowing  the  data  value  to  expire  could  prove  fatal. 
Consequently,  we  need  combinations  of  flexible  communication  schemes  as  part  of  the 
system  of  systems.  Such  a  system  should  exploit  data  gathered  from  networked  assets  in 
a  timely  manner  and  allow  for  the  extraction  of  vital  infonnation  in  real  or  near  real-time. 
Automating  specific  procedures  within  short  processing  times  can  increase  effectiveness, 
especially  for  cases  in  which  human  operators’  perfonnance  is  not  equivalent.  Artificial 
intelligence  (AI),  machine-learning,  and  data-mining  can  support  this  idea,  at  least 
starting  with  well-defined  procedures  concerning  structured  decisions  [1]  and  database 
exploitation.  The  more  sensors  we  are  able  to  operate  and  integrate  into  automated 
processes,  the  closer  we  will  be  to  real-time  infonnation  that  covers  any  spots  that  would 
otherwise  be  blind  due  to  lack  of  data  exchange  and  exploitation. 
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Databases  serve  as  assisting  mechanisms  for  human  decision-making  processes. 
Although  there  are  cases  in  which  improper  use  of  data,  based  on  lack  of  integrity,  might 
lead  to  wrong  assumptions,  in  general,  databases  facilitate  decision-makers.  In  our  case, 
and  for  the  scope  of  this  research,  it  is  impossible  for  a  human  operator  to  remember 
thousands  of  images  of  different  types  of  vessels.  Previous  work  from  ship  companies  has 
shown  that  creating  databases  from  such  kinds  of  data  accompanied  by  the  appropriate 
metadata  has  proven  crucial  for  recognizing  COIs.  The  use  of  such  data  during  operations 
has  demonstrated  significant  results  in  the  efforts  of  crews  for  target  recognition  by  type 
or  even  by  identity.  Especially  in  the  case  of  recognizing  targets  by  type,  the  effective 
distance  of  personnel  after  a  six-month  period  of  training  was  doubled  by  the  use  of  such 
data  [2],  The  downside  of  this  process  is  that  it  requires  significant  time,  resources,  and 
personnel. 

The  implementation  of  an  automated  process  that  is  specifically  focused  on  the 
recognition  of  ship  types  or  identities  could  prove  crucial  in  the  effort  to  increase  SA.  A 
similar  concept  that  was  implemented  in  the  areas  of  face  and  fingertip  classification  and 
matching  has  proven  beneficial.  Feineigle,  Morris,  and  Snyder  [3]  described  the  need  for 
automatic  target  recognition  (ATR)  systems  using  imaging  sensors  and  databases  as  well 
as  the  benefits  of  these  systems  for  maritime  domain  awareness  (MDA)  in  harbors.  Our 
research  further  expands  these  ideas  and  presents  the  challenges  of  recognition  in  larger 
areas  of  interest  (AOIs)  through  networking  kinetic  and  non-kinetic  platforms.  In 
addition,  we  consider  the  need  for  extending  these  capabilities  during  night  operations. 

Another  very  important  aspect  of  this  research  is  that  it  considers  implementing 
an  appropriate  level  of  automation  in  certain  tasks  and  procedures  to  increase  SA  and 
improve  the  decision-making  process.  Some  crucial  features  of  this  process  include 
flexible  networking  solutions  and  automated  asset  monitoring  to  capture  and  disseminate 
data;  these  considerations  are  especially  challenging  in  the  littoral  environment. 
Emphasis  is  also  given  to  the  automatic  exploitation  of  this  data  to  enable  filtering  and  to 
focus  only  on  targets  that  demonstrate  characteristics  of  potential  COIs.  Technological 
advancements  in  sensor-imaging,  processing  speed,  computer-vision,  and  machine- 
learning  seem  to  support  this  idea. 


2 


Based  on  these  concepts,  this  research — through  simulation  and  experimentation 
in  the  focus  areas  of  wireless  mesh  networks  (WMNs),  computer-vision,  and  machine- 
learning — emphasizes  the  apparent  need  for  adopting  new  tactics,  techniques,  procedures, 
and  doctrines  in  littoral  water  operations.  Proliferation  and  imaging  capabilities  of 
unmanned  systems  (UMSes)  and  small  satellites  are  factors  we  need  to  consider  through 
this  process.  Networking  and  exploiting  data  from  unmanned  assets  and  dispersed  sensors 
as  well  as  adopting  ATR  procedures  with  certain  levels  of  automation  can  increase  SA 
and  effectiveness  in  combat.  Then,  we  can  support  decisions  by  providing  the  decision¬ 
maker  timely  access  to  the  appropriate  infonnation. 

B.  RESEARCH  QUESTIONS 

The  overall  concept  is  a  vision  of  a  system  that  through  distributed  database 
centers  aboard  main  units  and  wireless  mesh  technologies  enhances  SA  and  reduces 
decision-making  cycles.  Distributed  management  of  littoral  operations  at  the  tactical  level 
should  provide  flexibility  and  alternatives  in  this  type  of  operation,  in  which  timeliness  of 
information  and  reduced  decision  cycles  are  of  critical  importance.  In  cases  where 
geography  permits,  such  as  in  Greece  and  the  Aegean  Sea,  this  concept  could  prove 
beneficial  in  a  form  of  a  system-of-systems  for  innovative  tactical  solutions  in  all  kinds  of 
military  and  homeland  security  missions.  All  the  participating  assets,  from  humans  to 
unmanned  systems,  could  become  part  of  an  ad-hoc  network  that  contributes  in  multiple 
ways  to  current  operations.  Some  of  these  solutions  could  include  agile  communication 
relays  and  planning,  extension  of  coverage  ranges  for  sensors,  distributed  lethality,  as 
well  as  video  and  image  streaming  from  multiple  dissimilar  sensors. 

Since  the  scale  of  such  a  concept  is  beyond  the  scope  of  a  single  thesis,  we  are 
focusing  the  research  in  two  main  categories  of  questions  that  contribute  to  proving  its 
feasibility.  The  first  category  of  question  is  related  to  the  capabilities  of  wireless  mesh¬ 
networking  and  its  potential  use  by  maritime  units  with  the  additional  functionality  of 
distributed  data  centers  and  network  operations  centers  (NOCs).  The  following  questions 
fall  under  this  category: 
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•  Can  we  achieve  flexible  and  robust  wireless  network  schemes  to  extend 
sensor  networks  connecting  human  or  machine  agents  on  terrestrial  or 
satellite  networks  in  littoral  waters? 

•  Is  it  possible  by  using  mesh  radio  technology  to  develop  and  manage  a 
robust,  ad-hoc,  and  seamless  network  to  facilitate  near  real  or  real-time 
data  dissemination  from  networked  imaging  sensors? 

The  second  category  of  question  involves  an  automated  process  that  takes 
advantage  of  distributed  imagery  databases.  Solutions  that  could  provide  for  ATR  and  the 
potential  benefits  of  applying  multi-sensor  image  fusion  are  explored.  The  following 
questions  are  relevant  to  this  category: 

•  Does  image  fusion  from  dissimilar  sensors  provide  better  classification 
performance  of  potential  targets  than  single  sensors? 

•  What  is  an  appropriate  scheme  to  facilitate  ATR  through  image  fusion  of 
networked  sensors? 

C.  HYPOTHESIS  AND  EXPLANATION 

We  attempted  to  find  answers  for  these  questions  by  performing  simulations  and 
experiments  as  well  as  by  capturing  data  in  the  field  from  multiple  dissimilar  sensors. 
First,  wireless  mesh  networks  were  simulated.  Then,  quality  of  service  (QoS)  in  terms  of 
provided  throughput,  bit  error  rate  (BER),  and  packet  losses  was  investigated.  Next,  field 
experiments  were  analyzed  to  support  our  simulation  results.  These  experiments  tested 
Wave  Relay  performance,  network-management  feasibility,  and  innovative  wireless¬ 
networking  solutions  for  imagery  data  dissemination.  Furthennore,  two  different 
cameras — both  with  color-visual  and  thermal  recording  capabilities — were  used  to 
capture  images  from  the  field.  Although  FLIR  C2  and  FLIR  SC640  cameras  had  similar 
capabilities,  the  different  analysis  techniques  for  the  images  posed  additional  challenges, 
similar  to  those  of  real  situations.  Finally,  fusion  of  the  acquired  images  was  evaluated  in 
our  ATR-implemented  process.  We  sought  to  propose  a  simulation  model  that  could 
work  well,  even  if  characteristics  of  the  images  were  different. 

Our  hypotheses  were  that 

•  QoS  provided  by  WMNs  can  accommodate  imagery  data  dissemination. 
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•  Fusion  of  images  from  dissimilar  sensors  will  provide  richer  features  to 
distinguish  between  different  classes  of  ships  in  a  highly  cluttered  and 
challenging  littoral  environment. 

In  the  case  that  our  hypotheses  were  proven  correct,  there  would  be  a  high  value  in 
finding  ways  to  extend  sensor  networks  that  could  contribute  to,  among  other  things,  the 
ATR  process  of  COIs  in  the  operational  environment  of  littoral  waters.  This  means  that 
we  had  to  prove  that  this  network  would  be  resilient  and  capable  of  transferring  in  a 
timely  manner  the  acquired  data  from  heterogeneous  platforms.  We  attempted  to  prove 
this  concept  by  simulating  the  conditions  and  parameters  of  equipment  that  have  already 
been  tested  through  field  experimentation.  We  also  considered  the  overall  benefits  that 
the  whole  concept  could  bring  to  the  operational  doctrine  of  littoral  waters  since  the 
proliferation  of  unmanned  systems  and  small  satellites  with  imaging  and  networking 
capabilities. 

D.  SCOPE  AND  LIMITATIONS 

This  thesis  sought  to  explore  the  feasibility  and  advantages  of  combining  the  data 
from  similar  or  dissimilar  sensors  aboard  platforms  in  and  out  of  mobile  ad-hoc  networks 
in  a  littoral  environment.  In  such  a  challenging  environment,  the  speed  and  accuracy  of 
information  obtained  plays  a  vital  role  in  the  decision-making  process  of  the  various 
missions  assigned  to  military  and  homeland  security  forces.  Inevitably,  we  sought  for  real 
or  near  real-time  data  transfer  and  exploitation  by  classifying  the  obtained  data  from  the 
available  sensors.  Our  intention  was  that  the  methods  and  processes  should  not  limit  the 
use  of  certain  sensors  but  help  find  ways  for  exploiting  the  available  data  for  fusion,  no 
matter  the  source. 

Fusion  methods  can  be  applied  in  several  ways  and  in  different  stages  of  such 
processes  while  the  sources  can  be  either  passive  or  active  sensors.  In  this  thesis,  only 
passive  means  for  obtaining  images  were  exploited,  specifically  for  fusion  of  color-visual 
and  thermal  sensors  in  early  or  late  stages  of  the  proposed  classification  process. 
Additionally,  we  limited  our  research  to  a  binary  classification  problem  of  distinguishing 
between  sailing  and  power  boats  by  type  from  the  images  captured  in  the  field.  This 
limitation  did  not  guarantee  that  the  techniques  used  would  be  directly  applicable  for 
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classifying  other  categories  of  objects  or  for  using  different  combinations  of  sensors. 
However,  special  care  was  taken,  so  the  collection  of  images  could  address  the  challenges 
of  classifying  ships  in  a  littoral  environment. 

Although  the  sensors  used  for  the  research  were  not  of  the  type  that  will  be  used 
in  a  tactical  environment,  the  transferring  capability  of  the  images  was  used  to  derive 
results  for  the  relative  research  questions.  The  simulations  of  the  mesh  network  were 
conducted  with  inputs  as  close  as  possible  to  equipment  already  in  use  while  the  Center 
for  Network,  Innovation  and  Experimentation  (CENETIX)  provided  the  opportunity  for 
evaluating  and  assessing  the  obtained  results. 

E.  RELATED  WORK 

Recent  advancements  in  technology  show  that  Cebrowski’s  vision  for  network¬ 
centric  warfare,  which  connects  not  only  platfonns  but  also  individual  fighters,  has 
become  a  reality.  This  revolution  in  military  affairs  (RMA) — combined  with  the 
proliferation  and  advancement  of  UMSes  that  are  replacing  human  agents  in  the 
battlefield — has  slowly  changed  not  only  how  but  more  importantly  who  fights,  as  argued 
by  Singer  [4]. 

This  evolution  has  no  meaning  if  we  are  unable  to  support  the  decision-making 
process  in  the  highly  mobile  and  fluid  environment  of  networks  formed  by  human  and 
machine  agents.  Iapichino  et  al.  proposed  a  hybrid  use  of  satellites  and  WMNs  to  support 
broadband  communications  in  areas  with  no  infrastructure  [5],  The  proposed  scheme 
offered  a  viable  solution  for  establishing  reliable  communications  between  various 
mobile  nodes  in  emergency  situations  and  was  based  on  the  IEEE  802.11s  standard  [6], 
[7],  [8].  Additionally,  Bordetsky  and  Dolk  introduced  the  concept  of  a  network  decision 
support  system  (NWDSS)  [9],  which  connected  available  nodes  through  wireless  ad-hoc 
networks  to  aid  the  decision-making  process.  Gerla,  Lau,  and  Oh  [10]  built  upon  the 
same  concept  by  proposing  the  use  of  content-centric  networking  (CCN)  when 
emergency  wireless  ad-hoc  networks  are  needed  on  the  battlefield.  A  hierarchical 
network  structure  model  in  the  battlefield  is  depicted  in  Figure  1.  In  this  way,  efficient 
network  resource  management — among  other  crucial  characteristics  such  as  security  and 
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scalability — was  achieved  for  mobile  ad-hoc  networks  (MANETs).  Especially  in  the 
complex  and  challenging  environment  of  littoral  waters,  the  aforementioned  evolutions 
are  providing  new  insight  into  updating  current  doctrine,  as  mentioned  by  Bordetsky, 
Benson,  and  Hughes  [11]. 

Figure  1 .  Structure  of  a  Hierarchical  Mesh  Network  Model  in  the  Battlefield 


This  structure  model  shows  how  group-based  mobility  can  facilitate  tactical  and 
emergency  operations.  Source:  Oh,  Lau,  and  Gerla,  “Content  Centric  Networking  in 
tactical  and  emergency  MANETs,”  2010. 


Distributed  network  operations  centers  (NOCs)  aboard  major  units  or  ground 
stations  are  considered  crucial  to  efficiently  managing  these  highly  kinetic  nodes  and  the 
data  acquired  in  the  field.  Experience  on  the  battlefield  as  well  as  research  and 
experiments  by  CENETIX  have  proven  that  using  such  applications  could  greatly 
enhance  SA.  Effort  in  this  direction,  including  the  CodeMettle  ecosystem  and  dashboard 
for  network  management  and  monitoring  (see  Figure  2),  has  demonstrated  very 
promising  results  [12],  [13]. 
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Figure  2.  Network  Operations  Ecosystem  and  Dashboard  from  CodeMettle 


The  ecosystem  and  dashboard  from  CodeMettle  demonstrates  how  network  management 
can  contribute  to  decision  making  by  converging  vital  information  in  a  dashboard  and 
applying  the  necessary  level  of  automation.  Source:  CodeMettle,  LLC,  “The  CodeMettle 
Ecosystem  for  Network  Management  in  support  of  the  DOD’s  NMS  convergence 
efforts,”  2015. 

Given  the  fluid  nature  of  these  fonned  networks  and  the  need  to  support 
traditional  decision-making  models,  such  as  Boyd’s  observe-orient-decide-act  (OODA) 
loop  [14],  [15],  new  challenges  have  emerged  in  monitoring  systems  since  the 
proliferation  of  unmanned  airborne  systems  (UASes)  and  small  satellites.  Spitzer, 
Kappes,  and  Boker  [16]  recognized  this  need  for  interoperability,  automatic 
dissemination,  and  exploitation  of  data  derived  from  systems  deployed  in  the  operational 
theatres.  Their  proposed  model  for  tasking,  collecting,  processing,  exploiting,  and 
disseminating  data  provides  insight  into  the  effective  fusion  of  data  obtained  from 
intelligence,  surveillance,  and  reconnaissance  (ISR)  missions  and  networked  assets.  Mesh 
networks  can  be  one  of  the  means  to  achieve  this  goal  as  they  provide  low  probability  of 
detection,  high  throughput,  the  flexibility  to  override  challenges  posed  in  littoral 
environments,  distributed  lethality  [17],  and  agile  emission  control  (EMCON)  [18]. 

In  such  a  complex  environment  as  the  modern  battlefield,  automating  procedures 
in  which  computers  perfonn  better  than  humans  is  necessary.  One  of  the  procedures  that 
contributes  to  SA  and  target  allocation  is  ATR.  Research  in  the  field  of  ATR  has 
produced  only  partial  solutions — with  certain  limitations — because  the  concept  is 
complex  and  requires  application  in  greatly  diversified  environments  and  conditions. 
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Several  attempts  have  been  made  to  design  and  develop  ATR  procedures  that 
could  support  classification  of  COIs.  Menon,  Boudreau,  and  Kolodzy  [19]  introduced 
such  a  process  to  classify  naval  targets  from  imagery  captured  by  inverse  synthetic 
aperture  radar.  Features  were  extracted  from  the  images  and  classified  using  neural 
networks.  Additionally,  Alves,  Herman,  and  Rowe  [20]  tried  another  approach  to  classify 
forward-looking  infrared  (FLIR)  images  of  ships  using  neural  networks  that  were  trained 
with  models  of  ship  classes  of  various  aspects  and  scale-invariant  moments  as  features.  In 
other  attempts  for  all-aspect  ship  recognition  using  IR  images,  the  importance  of  role, 
range,  and  aspect  angle  was  described  [21],  [22],.  Although  these  works  demonstrated 
good  results  of  more  than  seventy  percent  of  correct  classification,  image  processing 
methods  used  were  not  always  automated  [22],  and  the  training  and  testing  processes 
were  time  consuming  . 

Feineigle,  Morris,  and  Snyder  [3]  seemed  to  overcome  the  aforementioned 
problems  by  introducing  the  classification  features  of  scale  invariant  feature 
transfonnation  (SIFT),  local  interest  point  detection,  and  descriptors  of  optical  imagery. 
In  this  way,  features  that  compared  in  the  classification  process  were  invariant  to  scale 
and  rotation  and  partially  invariant  to  illumination  and  aspect.  Furthermore,  introduction 
of  speeded-up  robust  features  (SURF)  [23],  which  were  mainly  used  in  matching 
problems  in  computer  vision,  demonstrated  better  and  faster  results  in  the  feature- 
detection  process. 

Our  approach  in  the  proposed  ATR  process  tried  to  take  advantage  of  the  SURF 
characteristics  for  real-time  classification  of  ships.  Furthermore,  we  considered  that  any 
proposed  scheme  should  be  applicable  to  a  system  of  multiple  dissimilar  or  similar 
sensors.  For  this  reason,  different  cameras  were  used  for  the  collection  of  the  images  for 
this  experiment.  The  use  of  fused  multi-spectral  infrared  (IR)  and  visual  images  were  also 
tested — as  previous  work  has  proven  that  fusion  results  in  images  that  provide  additional 
information  even  at  night  [24]. 


9 


F.  SIMULATION  AND  EXPERIMENTATION  STEPS  AND  TOOLS 

The  method  we  used  in  this  research  was  quantitative  experimentation  using 
computer  simulations,  experiments,  and  data  collection  in  the  field  in  an  attempt  to 
simulate  the  conditions  and  stages  of  a  conceptual  model  for  ATR  through  mesh  tactical 
networking  of  participating  nodes.  Evaluation  and  analysis  of  results  from  field 
experiments  conducted  under  CENETIX  were  used  to  support  specific  parts  of  the 
concept.  The  steps  in  this  thesis  included  the  following: 

•  Simulating  a  wireless  hybrid  mesh  network  in  a  littoral  environment  using 
the  Systems  Tool  Kit  (STK); 

•  Analyzing  the  communication  links  by  studying  the  physical  layer  of  the 
network  in  STK; 

•  Analyzing  the  network  from  the  MAC  layer  and  above  with  QualNet; 

•  Evaluating  the  QoS  provided  by  the  network  to  support  video  streaming 
and  imagery  data  dissemination; 

•  Analyzing  CENETIX  field  experiments; 

•  Capturing  imagery  data  from  multiple  dissimilar  sensors  (visual  and 
thermal)  in  the  field;  and 

•  Evaluating  multi-sensor  image  fusion  for  ATR  by  using  computer- vision 
and  machine-learning  techniques. 

Various  software  tools  were  employed  to  meet  the  objectives  and  the  standards  of 
our  experiments  and  simulations.  Descriptions  and  characteristics  of  STK,  QualNet,  and 
MATLAB,  some  of  the  main  tools  utilized  in  this  thesis,  are  described  in  the  next 
subsections. 

1.  Systems  Tool  Kit 

STK  is  physics-based  software  released  by  Analytical  Graphics,  Inc.  (AGI).  Its 
simulation  capabilities  span  land,  sea,  air,  and  space  environments  where  various  assets 
and  attached  sensors  can  be  modeled  and  displayed.  The  software  automatically 
calculates  spatial  relationships,  such  as  line-of-sight  (LOS),  among  the  various  platforms 
and  nodes.  The  final  product  after  implementing  desired  properties  can  be  simulated  for 
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specified  time  intervals  in  both  two  and  three  dimensions  (2D  and  3D).  Additionally, 
images  and  video  can  be  recorded  during  the  simulations  with  the  available  tools  [25]. 


Figure  3.  Systems  Tool  Kit 


Adapted  from  “Systems  Tool  Kit  version  10,”  http://blogs.agi.eom/agi/2012/l  1/15/ 
systems-tool-kit-version-10-is-here/,  and  “System  Tool  Kit  -  Because  we  model  more 
than  just  satellites.,”  http://blogs.agi.com/agi/2012/07/25/svstems-tool-kit-%E2%80%93- 
because-we-model-more-than- just-satellites/ 

The  most  important  advantage  of  STK  is  the  variety  of  missions  and  systems  that 
it  can  simulate.  STK  can  even  help  the  user  simulate  and  evaluate  a  system-of-systems 
and  its  desired  or  researched  topology.  Some  modules  that  interoperate  with  STK  to 
evaluate  the  simulations  include  MATLAB,  QualNet,  and  ArcGIS.  These  additional 
modules  contribute  to  a  more  thorough  analysis  and  detailed  simulation.  After  detailed 
parameters  or  snippets  of  code  are  imported  into  STK,  data  elevation  models  (DEMs)  and 
maps  can  be  imported,  and  the  network’s  performance  can  be  examined  beyond  link 
budgets. 
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Figure  4.  STK  Objects 


The  STK  objects  available  as  they  appear  on  the  relevant  pop-up  window  when  insert  of 
a  new  object  is  selected. 


STK  version  11,  whose  release  is  planned  for  January  2016,  could  contribute  in  a 
more  thorough  analysis  for  our  simulation.  Volumetric,  EO/IR,  and  aviator  capabilities, 
which  will  be  part  of  the  new  version,  could  provide  extra,  useful  details  in  our  analysis 
above  mesh  network  one. 

2.  QualNet 

The  QualNet  interface  is  one  of  the  off-the-self  products  that  can  interoperate  with 
STK  as  an  add-in  module.  It  offers  a  network-modeling  platform  to  plan  and  test  a 
communication  network’s  performance  while  taking  advantage  of  the  various  kinetic  and 
non-kinetic  models,  linking  budget  calculations,  and  augmenting  visualization  of  STK. 
Hence,  its  use  offers  a  more  complete  illustration  of  how  network  performance  affects 
overall  mission  planning  and  execution  [26],  [27].  This  interface  allows  the  incorporation 
of  all  STK  constraints,  antennas,  propagation  loss,  atmospheric  absorption,  as  well  as 
terrestrial  and  custom  models  for  the  purposes  of  network  analysis.  The  exchange  of  data 


12 


between  the  QualNet  and  the  STK  model  and  objects  results  in  a  more  thorough  analysis 
by  combining  the  advantages  of  both  tools.  During  the  QualNet  analysis  and  whenever  a 
communications  link  is  established  between  the  wireless  nodes,  the  combined 
interactions  and  inputs/outputs  are  dynamically  generated  (see  Figure  5). 


Figure  5.  QualNet/STK  Interaction  Flow 
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Source:  “STK  10  Help,”  http://www.agi.eom/resourees/help/online/stk/10.l/ 


The  procedure  to  install  and  integrate  the  Qualnet  add-in  in  STK  proved  rather 
tedious  but  was  finally  achieved  after  following  the  steps  described  in  QualNet  and  STK 
communication  troubleshooting  and  installation  and  in  a  suggestion  by  Schlatow  and 
Lefherz  [28].  The  steps  to  run  QualNet  and  obtain  its  results  after  successful  integration 
are  shown  in  Figure  6. 
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Figure  6.  QualNet/STK  Interaction  Flow 
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Source:  “STK  10  Help,”  http://www.agi.eom/resourees/heto/online/stk/10.l/ 


3.  MATLAB 

MATLAB  is  a  high-level  language  that  offers  a  variety  of  toolboxes  and  an 
environment  for  programming,  numerical  computation,  and  visualization  of  processed 
data.  MATLAB’ s  built-in  functions  and  tools  provide  support  for  a  wide  range  of 
domains,  from  signal  processing  to  aerospace.  In  this  research,  image  processing, 
computer-vision,  as  well  as  the  neural  network  (NN)  toolbox  were  used  to  build  an 
algorithm  for  the  ATR  of  boat  images  captured  by  color- visual  and  thermal  sensors. 


Figure  7.  MATLAB  Logo 
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Source:  “Image  Analysis  with  MATLAB,”  http://videreanalytics.ca/events/image- 
anal  vsis-with-mat  lab-mav-20-20 1 5/ 
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G. 


ORGANIZATION  OF  THE  THESIS 


This  thesis  is  organized  in  five  chapters  in  an  effort  to  present  and  tie  together 
different  experiments  and  simulations  for  using  mesh  tactical  networks  of  sensors 
efficiently  for  ATR.  In  Chapter  II  we  present  a  brief  theoretical  background  on  mesh 
networks  and  relevant  technologies,  imaging  sensors,  and  digital  image  representation,  as 
well  as  the  techniques  and  algorithms  used  in  the  image  classification  process.  A 
proposed  simulation  model  for  the  ATR  through  a  hybrid  mesh  network  is  proposed  in 
Chapter  III  along  with  a  thorough  explanation  of  the  steps  that  were  followed  for  the 
experiments  and  the  simulations.  The  results  derived  from  this  research  and  their  analysis 
are  presented  in  Chapter  IV.  In  Chapter  V,  conclusions  from  the  results  and  potential 
benefits  in  the  modem  battlefield  are  discussed  as  well  as  recommendations  for  future 
work  in  the  area  of  our  research. 
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II.  THEORETICAL  AND  TECHNICAL  BACKGROUND 


This  chapter  provides  a  brief  background  of  the  equipment,  concepts,  techniques, 
and  procedures  used  in  the  simulations  and  experiments  for  accomplishing  this  research. 

A.  SIMULATING  MANET  AND  MESH  NETWORKS  IN  MARITIME 

LITTORAL  ENVIRONMENT 

Simulation  has  been  proven  highly  beneficial  as  it  can  support  and  assist  in  the 
thorough  and  detailed  planning  of  an  exercise  or  experiment  with  a  profound  cost  benefit. 
Additionally,  while  resources  are  often  limited,  simulations  can  provide  insight,  results, 
and  evaluation  for  large-scale  scenarios.  In  the  case  of  mobile  ad-hoc  networks 
(MANETs)  and  wireless  mesh  networks  (WMNs),  simulations  can  be  used  to  predict 
performance.  Furthermore,  they  can  calculate  or  provide  insight  into  the  necessary  link 
margins  or  antenna  gains  needed  to  improve  the  perfonnance  of  certain  links.  Finally, 
when  simulation  results  are  compared  with  those  from  actual  field  experiments  or  real 
operations,  they  can  provide  patterns  and  insight  for  further  research  and  development  of 
new  ideas  and  techniques. 

1.  Wireless  Networks 

Wireless  networks  are  based  on  various  technologies  regulated  by  IEEE 
standards.  Although  they  can  provide  short-distance  communications  with  technologies 
like  Bluetooth,  wireless  networks  can  also  support  long-range  communications  up  to  30 
miles  or  more  depending  on  the  antennas  used.  The  most  popular  wireless  technologies 
are  Wi-Fi  and  WiMAX,  under  IEEE  802.1 1  and  IEEE  802.16  standards,  respectively.  Via 
mobile  WiMAX,  kinetic  nodes  can  connect  with  broadband  communication  channels 
using  spread  spectrum  techniques,  such  as  orthogonal  frequency  division  multiplexing 
(OFDM),  and  can  achieve  data  rates  of  up  to  one  gigabit  (Gbit)  per  second  (s)  [6].  The 
quality  of  service  (QoS)  provided  can  simultaneously  support  video  streaming  and  voice 
traffic  over  the  same  channel  [29].  Under  the  IEEE  802.11s  standard,  peer-to-peer  nodal 
relationships  for  mesh  networking  allow  for  self-forming  and  self-healing  characteristics 
[8].  In  Figure  8,  a  basic  mesh  topology  is  depicted  as  well  as  the  possible  mesh  paths. 


17 


Figure  8.  Basic  Mesh  Topology 


Source:  [8]  J.  Henry  and  M.  Burton,  “802.11s  Mesh  Networking,”  CWNP  Enterprise  Wi¬ 
Fi  Whitepaper,  November  2011. 


a.  MANETs 

MANETs  are  networks  formed  by  devices  that  communicate  with  each  other 
within  link  range  and  bandwidth  constraints,  without  the  underlying  need  of  any 
infrastructure,  base  stations,  or  network  configuration.  Link  range  depends  greatly  on  the 
frequency  used  and  the  rest  of  the  characteristics  that  form  the  wireless  transmissions,  for 
example,  RF  modulations  or  power  output.  Each  mobile  node  of  these  networks  acts  as 
both  a  terminal  and  a  router  that  forwards  any  received  data  to  the  rest  of  the  nodes 
within  range.  Obviously,  determining  which  nodes  are  contributing  to  the  network  is  a 
highly  dynamic  procedure  as  MANETs  are  decentralized  networks.  For  this  reason,  all 
network  activity,  topology  discovery,  and  message  delivery  are  individually  controlled  by 
each  node  [7],  [30],  [6].  Types  of  nodes  that  can  be  networked  in  specific  applications  are 
shown  in  Figure  9. 
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Figure  9.  Assets  and  Agents 


(b) 


Examples  of  assets  and  agents  that  can  be  networked  through  ad-hoc  networks  in  military 
(a)  or  Humanitarian  assistance  and  disaster  relief  (HA/DR)  applications  (b).  Source:  [31] 

David  Munoz,  Frantz  Bouchereau  Lara,  and  Cesar  Vargas,  “Ad  hoc  and  Sensor  Network 
Scenarios,”  2011,  http://www.eetimes.com/document.asp7doc  id=1279149&. 

Different  techniques  and  protocols  can  be  used  for  forwarding  and  relaying  data 
and  packets  via  MANETs.  During  routing,  the  messages  propagate  along  an  existing 
detennined  path,  hopping  from  node  to  node  until  they  reach  the  intended  destination.  In 
case  a  portion  of  the  path  breaks  during  the  hopping  process,  the  network  finds  a  way 
around — illustrating  its  self-healing  characteristic — via  the  remaining  available  links. 
Determining  viable  and  robust  paths  in  such  networks  is,  therefore,  a  great  challenge.  In 
certain  cases,  packet  losses  inevitably  occur  [31].  A  variety  of  factors  influences  these 
highly  mobile  and  fluctuating  networks,  and  routing  protocols  have  to  compensate  for 
any  of  these  issues.  Continuous  and  unpredicted  changes  in  network  topology,  power 
constraints,  wireless  link  quality,  path  losses,  and  interference  are  some  of  these 
factors  [30]. 
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Three  categories  of  routing  are  used  by  mobile  routers  in  wireless,  multi-hop 
networks  such  as  MANETs  and  WMNs.  These  include  table-driven,  or  proactive, 
routing;  on-demand,  or  reactive,  routing;  and  hybrid  routing,  a  combination  of  the  two 
former  categories  [32].  There  is  one  primary  difference  between  the  two  main  categories 
of  routing.  In  proactive  routing,  there  is  a  requirement  that  routing  information  is  stored 
by  all  routers,  even  for  those  that  are  inactive.  Conversely,  in  reactive  routing,  only  the 
routing  information  from  active  nodes  requires  storing  [32],  [33],  [34].  Because  each 
node  in  mobile  ad-hoc  networks  acts  simultaneously  as  a  host  and  a  router  and  due  to  the 
continually  changing  nature  of  nodes,  hybrid  protocols  often  provide  better 
performance  [35], 

Despite  the  problems  that  these  networks  face,  their  use  is  rapidly  proliferating, 
primarily  because  existing  applications  offer  flexibility  and  reasonable  ranges  and 
capacities.  Thus,  MANETs  and  WMNs  can  be  used  in  a  variety  of  cases  when  the 
underlying  infrastructure  requires  extension,  it  has  been  destroyed,  or  it  does  not  exist  at 
all.  Humanitarian  assistance  and  disaster  relief  (HA/DR)  as  well  as  military  operations 
are  such  cases.  These  circumstances,  especially  military  operations,  require  low 
probability  of  interception  (LPI),  high  reliability,  low  latency,  and  resolution  of  security 
issues,  all  need  to  be  achieved  [30]. 

b.  Wireless  Mesh  Networks 

WMNs  are  a  specific  type  of  the  aforementioned  MANETs.  Therefore,  WMNs 
inherit  the  MANET  attributes  of  self-discovery,  self-organization,  self-configuration,  and 
self-healing  [7].  The  difference  that  appears  in  WMNs  is  that  part  of  the  nodes  can 
remain  stationary  or  carry  advanced  capabilities.  Therefore,  there  is  the  potential  to 
interconnect  with  existing  networks  through  gateways  that  coexist  with  the  mesh  routers. 
The  mesh  in  WMNs  directly  relates  to  the  fully  connected  topology  shown  in  Figure  10. 
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Figure  10.  Network  Topologies 
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Source:  “Network  topology,”  http://homepages.uel.ac.uk/ii0322392/top.html. 


In  actual  implementation,  though,  a  fully  connected  mesh  is  difficult  to  maintain 
as  these  networks  are  characterized  by  high  mobility  that  involves  nodes  dynamically 
entering  and  exiting  the  network.  Nevertheless,  the  participation  of  nodes  as  Internet 
gateways  (IGWs)  shows  great  potential  for  the  formation  of  extendible  and  highly  kinetic 
broadband  Internet  networks  in  locations  where  no  infrastructure  is  present.  IGWs  can 
provide  interconnection  among  such  networks  and  existing  terrestrial,  satellite,  or  mobile 
Internet  networks,  such  as  3G,  4G,  or  even  5G,  and  may  extend  connectivity  in  areas  out 
of  coverage  (see  Figure  1 1).  Moreover,  the  use  of  low  earth  orbit  (LEO)  satellites  in  mesh 
networks  can  reduce  latency  in  medium-  and  long-distance  communications  [36]. 
Additionally,  connectivity  without  Internet  access  can  be  maintained  between  the 
networked  assets,  even  when  the  infrastructure  malfunctions  or  the  IGWs  are  out  of 
reach. 
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Figure  11.  Samsung’s  Global  Internet  Access  Satellite  Network 
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Samsung’s  proposal  involving  4,600  low  earth  orbit  (LEO)  satellites  for  global  Internet 
coverage.  Adapted  from  D.  Borghino,  “Samsung’s  giant  satellite  network  could  enable 
high-speed  Internet  access  across  the  globe,”  2015,  http://www.gizmag.com/samsung- 
satellite-network-lnternet-access/38971/ 


c.  Operational  Advantages  of  Wireless  Mesh  Tactical  Radios 

The  operational  advantages  gained  from  the  use  of  mesh  tactical  radios,  as  proven 
in  several  CENETIX  field  experiments,  are  briefly  described  below: 

•  No  infrastructure  is  needed. 

•  Immediate  interoperability  between  platforms  of  different  branches  with 
no  other  equipment  is  common. 

•  High  throughput  exists  to  accommodate  simultaneous  video,  data,  and 
voice  communications. 

•  They  are  easy  to  transfer  in  the  field. 
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•  Versions  of  man-portable  devices  allow  for  human  agent  nodes  as  part  of 
the  network. 

•  Security  is  embedded. 

•  They  transmit  low  power  and,  thus,  LPI. 

•  They  allow  for  legacy-radio  tethering. 

From  all  the  advantages  that  such  systems  demonstrate,  we  see  that  mesh  tactical  radios 
are  appropriate  for  teaming  up  small  ad-hoc  groups  of  highly  kinetic  manned  or 
unmanned  systems  and  human  agents  in  the  battlefields.  Communications  of  this  type  can 
facilitate  agile  EMCON  plans  and  offer  ad-hoc  solutions  when  traditional  means  of 
communication  are  unavailable  or  deemed  inappropriate  for  tactical  use. 

2.  Wave  Relay  MANET  Nodes 

Nodes  that  are  connected  through  the  Wave  Relay  MANET  become  part  of  a 
secure  mesh  network.  The  patented  wireless  networking  protocol  by  Persistent  Systems, 
LLC  supports  the  connectivity  of  kinetic  and  non-kinetic  nodes  as  well  as  the  exchange 
of  real-time  data,  video,  voice,  and  other  applications  [37].  The  following  paragraphs 
present  the  capabilities  of  these  radios  (see  Figure  12)  as  the  simulations  center  on  their 
performance  in  a  littoral  environment. 


Figure  12.  Wave  Relay  MANET  Radio  Solutions 


Quad  radio  router  and  man  portable  units  Gen3  and  Gen  4  Source:  “Wave  Relay  MANET 
RadioSolutions.”http://www.oceano  logvinternational.com/en/Exliibitors/19 1060/Steatite- 
Ltd/Product  s/666963/Wave-Relav-MANET-Radio-Solutions 
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a. 


Wave  Relay 


The  Wave  Relay  system  is  a  peer-to-peer  wireless  MANET  network  in  which  no 
server  or  master  node  exists.  The  nodes  also  serve  as  routers  that  establish  and  maintain 
the  network  even  if  one  of  them  fails  to  communicate.  The  implemented  techniques 
increase  multicast  reliability,  maximize  capacity  over  the  RF  spectrum  under  use,  and 
minimize  network  control  requirements.  Unlimited  devices  can  be  part  of  the  resulting 
highly  scalable  network  as  there  is  no  hop  limit.  This  rapid  self-forming  and  self-healing 
network  operates  at  the  data-link  layer,  which  is  the  second  layer  of  the  Open  Systems 
Interconnection  (OSI)  seven-layer  model,  as  shown  in  Figure  13  [6]. 


Figure  13.  The  ISO/OSI  Model 
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Source:  “The  Open  Systems  Interconnection  (OSI)  seven-layer  model  standardized  by  the 
International  Standardization  Organization  ISO,”  https://commons.wikimedia.org/wiki/ 
File:Osi-model.png. 


Currently,  the  fifth  generation  of  Wave  Relay  radios  is  ready  for  release  and 
provides  more  advanced  capabilities  than  the  fourth  generation  that  has  already  been  used 
in  CENETIX  field  experiments.  For  shared  access  coordination,  the  random  access 
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protocol  of  carrier-sense  multi-access  with  collision  avoidance  (CSMA/CA)  is  used  as 
the  basis  for  the  wireless  networks  [6],  [37].  Wave  Relay  uses  a  3x3  multiple-input 
multiple-output  (MIMO)  technology  and  delivers  high  throughput  of  up  to  150 
megabytes  per  second  (Mbps)  in  extended  ranges.  It  is  IPv4  and  IPv6  compatible  and 
includes  a  server  with  an  integrated  dynamic  host  configuration  protocol  (DHCP)  [6]. 
The  embedded  cloud-relay  technology  makes  it  possible  to  bridge  beyond-line-of-sight 
(BLOS)  with  line-of-sight  (LOS)  networks  using  the  network  layer,  the  third  layer  of  the 
OSI  seven-layer  model,  techniques,  and  protocols.  In  this  way,  distributed  groups,  as 
shown  in  Figure  14,  smoothly  participate  in  the  network  as  if  locally  connected  via 
Internet  or  private  networks.  Table  1  presents  the  basic  specifications  of  Wave  Relay 
along  with  some  imported  parameters  from  Systems  Tool  Kit  (STK)  to  simulate  objects. 


Figure  14.  Cloud  Relay  Groups 


Source:  “Cloud  Relay,”  http://www.persistentsvstems.com/pcrsistent-svstems-cloud- 
relay/. 
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Table  1.  Wave  Relay  Specifications 


3*3  MIMO 

TECHNOLOGY 

Extended  Range 

1  SO  Mbps  of  throughput 

Maximal  Ratio  Combining 

Spatial  Multiplexing 

Antenna  Detection 

MODULAR  RADIO 

RF  Modulations:  OFDM  (64-QAM,  16-QAM,  QPSK,  BPSK) 

6W  Transmit  Power* 

Swappable  Radio  Modules/Radio  Boards 
•RF-100G/RF-1 100  -  L-Band  Frequency  Range:  1350  - 1390  MHz* 
•RF-2000/RF-2 1 00  -  S-Band  Frequency  Range;  2200  -  2500  MHzt 
•RF-300Q/RF-3100  -  C-Band  Radio  in  Development 

Software  configurable  bandwidths:  2.5  MHz,  5  MHz,  10  MHz,  20 

MHz,  40  MHz 

TX/RX  Operating  Modes:  All  modes  from  SISO  to  3x3  MIMO 

RolP 

Legacy  Radio  Tethering 

Legacy  Radio  Detection 

USB  Host /RS-232  Serial 

VIDEO 

HD-BNC  Connection 

36-SDI  &  Composite  Input 

Integrated  HD  H.264  Video  Encoding/Decoding 

GPS 

3.3V  Active 

Situational  Awareness 

Cursor-on-Target  Compliant 

1  Second  Updates 

PTT/EUD 

Dual  Active  PTT  Channels 

End  User  Device  (EUD)  Data  /  Charging 

USB  Host /RS-232 Serial 

HD  Video  Output 

DATA 

USB  On-the-Go 

RS-232  Serial 

Ethernet 

HD  Video  Input 

BATTERY 

Low  Battery  Alert 

Standard  Twist-Lock  Connector 

NETWORKING 

Advanced  multicast  algorithms 

Seamless  Layer  2  network  connectivity 

Integrated  serial-to-Etbemet  capability 

Cloud  Relay™ 

DLEP  Certified 

IPv4  and  IPv6  compatible 

Integrated  DHCP  server 

USB  RNDIS  Host  and  Device 

SECURITY 

Integrated  Hardware  Cryptographic  Acceleration 

CTR-AES-256  Encryption 

HMAC-SHA-256  Authentication  &  Integrity 

Utilizes  Suite-B  Algorithms 

Cryptographically  authenticated  Over-the-Air  Rekey  and  Key  Zero 
FIPS  140-2  Level  1 

Rechargeable  30  Day  Hold-Up  for  Keys  and  Configuration  Settings 

PHYSICAL 

1  £  x  2.6  x  4.7  inches  /  38. 1  mm  x 66mm  x  1 1 9.4mm  (NO  BATTERY) 

SPECIFICATIONS 

1  £  x  2.6  x  7.9  inches  /  38. 1  mm  x  66mm  x  200.7mm  (WITH  BATTERY) 

1 73  oz  /  490.5  grams  (NO  BATTERY) 

29.6  oz/ 839.1  grams  (WITH  BATTERY) 

MANUFACTURING  & 

ISO  900 1:2008  certified 

ENVIRONMENTAL 

Manufactured  to  MIL-STD  specifications 

IP68 

Coated  to  meet  MIL-C-64159  /  MIL-C-53039A(2)  standards 

Operating  temperature:  -40"  to  85’  C  /  -40’  to  1 85’  F 

Source:  “Wave  Relay  5  Integration  Unit,”  http://www.mgged-systems.com/products/ 
radio-communications/manet-radio-solutions/wave-relav-5.html. 
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3.  Types  of  Nodes 

The  types  of  nodes  that  can  be  part  of  a  MANET  or  WMN  are  numerous  and  can 
extend  from  single  human  agents  to  unmanned  systems  and  satellites.  Some  main 
characteristics  and  capabilities  of  the  most  important  nodes  are  described  in  the  following 
paragraphs.  Satellites  and  unmanned  systems  can  prove  highly  beneficial  for  these  types 
of  networks.  They  can  assume  multiple  roles,  acting  as  relays  to  extend  the  network  and 
also  as  platfonns  for  imaging  or  other  types  of  sensors. 

a.  Small  Satellites 

In  general,  depending  on  their  orbits,  satellites  fall  under  the  categories  of  low 
earth  orbit  (LEO),  medium  earth  orbit  (MEO),  geosynchronous  or  geostationary  earth 
orbit  (GEO),  and  highly  elliptical  earth  orbit  (HEO)  [38].  Furthennore,  if  we  consider  the 
actual  size  of  the  spacecraft,  satellites  are  also  categorized  as  large,  medium,  or  small,  as 
shown  in  Table  2.  During  the  last  few  years,  there  has  been  a  focus  on  research  and 
development  of  small  satellites.  Cube  satellites,  or  Cubesats,  usually  with  a  mass  of  less 
than  100  kilograms  (kg),  are  very  promising  because  of  their  small  size  and  low 
manufacturing  cost.  An  advantage  comes  from  their  potential  to  be  launched  in  groups 
from  platforms  other  than  the  traditional  rockets — and  thus  readily  available. 

Table  2.  Satellites  Categories  Based  on  Size 


Satellite 

Category 

Net  Weight 

Large 

>  1000  kgr 

Medium 

500  -  1000  kgr 

Mini 

100-  500  kgr 

Micro 

10  -100  kgr 

SMALL 

Nano 

SATELLITES 

Pico 

0.1  -  1  kgr 

Femto 

<  100  gr 

Satellite  categorization  according  to  their  net  weight  and  size.  Source:  A.  Bordetsky, 
“Micro  and  Pico  Satellites  in  Maritime  Interdiction  Operations,”  15th  Int.  Command 
Control  Res.  Technol.  Symp.  Track  2  Networks  Netw,  2010. 
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Cubesats  in  particular  have  a  volume  equal  to  one  liter,  or  ten  cubic  centimeters 
(cm3)  or  1U.  They  can  also  be  combined  to  form  bigger  satellites  by  attaching  multiple 
units  together.  In  that  way,  2U  or  multiple-U  satellites  can  be  constructed.  Additionally, 
there  are  several  kits  available  on  the  market  that  allow  a  do-it-yourself  construction,  as 
shown  in  Figure  15.  Of  course,  their  small  size  and,  consequently,  their  payload  limit 
their  capabilities  in  terms  of  power,  sensors,  or  antennas  that  can  be  mounted  to  them. 
Problems  such  as  less  precise  orbital  fidelity,  limited  orientation  control,  and  imprecise 
pointing  also  limit  their  capabilities  and  make  them  prone  to  becoming  debris.  Since  they 
are  positioned  in  LEO,  though,  they  need  less  power  and  demonstrate  smaller  latencies  in 
communication.  Other  factors  that  should  be  considered  are  the  revisit  time,  the  coverage, 
the  lifetime  expectancy,  the  number  and  position  of  the  ground  stations,  and  the  altitude 
within  the  range  of  LEO,  which  is  roughly  200  to  2000  kilometers  (km)  [38]. 


Figure  15.  Cubesats 


In  (a)  a  home  kit  framework  of  a  Cubesat  is  shown.  Source:  “Begin  your  CubeSat 
Mission  with  the  CubeSat  Kit,”  http://www.cubesatkit.com/.  In  (b)  a  3U  Cubesat  and  in 
(c)  a  6U  Cubesat  are  presented.  Source:  “CubeSat,”  http://space.skyrocket.de/doc  sat/ 
cubesat.htm. 
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For  the  purpose  of  our  STK  scenario,  we  simulated  the  capabilities  of  an  existing 
satellite.  A  Pico-satellite  for  Remote-sensing  and  Innovative  Space  Missions  (PRISM), 
shown  in  Figure  16,  was  chosen.  Its  optical  characteristics  are  presented  in  Table  3.  The 
extensible  boom,  shown  in  Figure  17,  allows  for  a  ground  resolution  of  10  meters  (m)  per 
pixel  from  the  800  km  altitude  of  its  actual  orbit.  We  altered  our  simulation  parameters, 
such  as  orbit  altitude  and  communication  capabilities,  to  serve  our  simulation  purpose  of 
providing  imagery  data  with  an  adequate  ground  resolution  to  detect  small  boats. 

Figure  16.  PRISM  Nanosatellite 


Theoretical  illustration  of  the  deployed  PRISM  nanosatellite  in  LEO  and  its  extensible 
boom  before  and  after  it  has  been  deployed  Source:  “PRISM,”  https://directory. 
eoportal.org/web/eoportal/satellite-missions/p/prism 


Figure  17.  PRISM’s  Boom 


Components  of  the  PRISM’s  extensible  boom.  Source:  “PRISM,”  https://directory. 
eopo  rtal.org/web/eopo  rtal/satellite-missions/p/prism. 
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Table  3.  PRISM  Optics 


Parameter 

Value 

Aperture  of  Lens 

lOOnmi 

Focal  Length 

800mm 

Wavelength  (Min.) 

380mn 

Wavelength  (Max.) 

700nm 

Rayleigh  Limit 

6.8/mi 

Pixel  Size 

4.7/im 

Number  of  Pixels 

1280x1024  (SXGA) 

Size  of  Image  Sensor 

S.5Sx6.S6nmi 

Ground  Resolution 

13.4m  (theoretical) 

Exposure  Time 

1 ,0msec 

PRISM  optics  specifications.  Source:  “PRISM,”  https://directory.eoportal.org/web/ 
eoportal/ satellite-missions/p/prism. 


b.  Unmanned  Systems 

The  use  of  unmanned  systems  is  rapidly  expanding  in  the  commercial  sector  as 
well  as  in  the  armed  forces.  Various  missions  are  assigned  to  such  systems  depending  on 
their  operational  environments.  Unmanned  airborne  systems  (UASes),  in  particular,  can 
potentially  extend  the  capabilities  of  their  mother  platforms  when  deployed  and 
controlled  by  them.  Data  that  is  fed  back  as  well  as  network  extension  in  the  case  of 
MANETs  provide  additional  options  and  capabilities  to  enhance  situational  awareness 
(SA)  on  the  battlefield.  Furthermore,  throwable  and  expendable  nodes  can  also  facilitate 
that  purpose. 


c.  Other  Nodes 

As  discussed  previously,  MANETs  and  WMNs  can  network  various  nodes  and 
extend  existing  terrestrial  or  satellite  networks.  This  concept  can  be  used  as  an  advantage 
especially  in  littoral  environments  to  network  manned  or  unmanned  platforms,  ground 
stations,  and  distributed  link  relays.  Underwater  systems  or  human  agents  could  also 
become  part  of  the  network  using  floating  platforms  and  wireless  technologies  as  recently 
demonstrated  by  CENETIX  experiments  in  the  San  Francisco  Bay  area  in  October  2015. 


30 


In  this  way,  special  and  boarding  operations  can  take  advantage  of  MANETs  to  exchange 
images  and  messages,  even  with  divers. 

4.  Network  Operations  Centers 

Network  operations  centers  (NOCs)  or  network  management  centers  are  specific 
locations  from  which  management,  control,  supervision,  and  monitoring  of  networks  can 
be  performed  [39].  The  goal  of  this  function  is  to  support  uninterrupted  and  robust 
accessibility  in  networked  resources  and  applications  to  all  subscribed  assets  by 
efficiently  managing  the  available  resources  and  security  issues.  Internet  service 
providers  (ISPs)  operate  such  centers  to  enable  visualization  of  the  controlled  and 
monitored  networks  as  well  as  their  statuses. 

a.  Network  Decision  Support  Systems 

The  concept  of  a  network  decision  support  system  (NWDSS)  was  introduced  by 
Bordetsky  and  Dolk  [9]  to  support  agile  and  collaborative  management  of  decision 
making  when  heterogeneous  nodes  are  connected  through  wireless  technologies.  This 
concept  offered  a  solution  for  enhanced  decision  making  and  SA  in  the  modem 
battlefield.  Since  the  proliferation  of  unmanned  systems  over  the  last  fifteen  years,  this 
idea  needs  revision  to  address  the  challenges  of  highly  kinetic  platforms,  especially  in 
littoral  waters.  Timeliness  in  such  environments  is  crucial,  and  the  proposed  sense- 
analyze-adapt-memory  decision-making  loop,  shown  in  Figure  18,  offers  a  team-centric 
solution. 
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Figure  18.  NWDSS  Decision-making  Loop 


The  Sense  -  Analyze  -  Adapt  -  Memory  decision  making  loop.  Source:  Bordetsky  and 
D.  Dolk,  “A  Conceptual  Model  for  Network  Decision  Support  Systems,”  Syst.  Sci. 
(HICSS),  2013  46th  Hawaii  Int.  Conf,  pp.  1212-1221,  2013. 


B.  IMAGING  SENSORS 

Imaging  sensors  are  widely  used  by  armed  forces  in  many  applications.  Various 
platforms  as  well  as  individual  soldiers  have  been  equipped  with  sensors  covering  a  wide 
range  of  the  visible  and  infrared  (IR)  spectrums.  What  led  to  this  proliferation  was  the 
realization  that  these  sensors  were  beneficial  for  SA  enhancement  and  target  recognition. 
During  the  day,  they  extend  the  capabilities  of  human  eyes  as  well  as  support  the  ability 
of  extracting  additional  information  about  the  state  of  targets.  For  example,  during 
daylight  hours,  IR  sensors  can  provide  infonnation  about  operating  equipment  or 
incoming  missiles  as  they  capture  thennal  signatures  otherwise  undetectable  by  sensors 
operating  in  the  visible  range  of  the  electromagnetic  (EM)  spectrum.  Furthennore,  during 
nighttime  hours,  IR  sensors  reveal  thermal  signatures  when  color-visual  sensors  become 
unusable.  In  the  following  subsections,  we  will  briefly  present  the  basic  concepts  and 

technologies  used  to  capture  and  support  the  exploitation  of  that  information. 
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1.  The  Electromagnetic  Spectrum 

To  provide  a  better  understanding  of  the  differences  between  visible  and  IR 
imagery,  we  begin  by  investigating  the  EM  spectrum.  As  Figure  19  shows,  the  visible 
region  is  only  a  small  part  of  it  the  entire  spectrum.  Photoreceptors,  or  cones,  of  the 
human  eye  are  sensitive  only  to  colors  in  the  visible  spectrum  [40]  covering  wavelengths 
380-780  nanometers  (nm)  [41]  . 

While  humans  can  only  feel  IR  radiation  in  the  form  of  heat,  animals  such  as 
rattlesnakes  can  actually  see  that  part  of  the  spectrum.  Using  two  different  sensor  types, 
one  operating  in  the  visible  and  the  other  in  the  IR  region,  it  is  possible  to  gather  more 
information,  depending  on  the  prevailing  conditions. 


Figure  19.  The  EM  Spectrum  Showing  Visible  and  IR  Bands 
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Based  on  atmospheric  transmittance  windows,  discussed  in  the  following 
subsection,  the  IR  spectrum  is  further  divided  into  near  (NIR),  short-wavelength  (SWIR), 
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mid-wavelength  (MWIR),  long-wavelength  (LWIR),  and  very  long- wave  length 
(VLWIR),  or  far  (FIR),  infrared. 

2.  EM  Radiation 

Incident  radiation  can  be  transmitted,  reflected,  scattered,  or  absorbed,  depending 
on  the  properties  of  the  material  it  encounters,  the  frequency  of  the  radiation,  and  the 
incident  angle  [42],  as  illustrated  in  Figure  20. 

Figure  20.  EM  Radiation  Interaction  with  Materials 


Source:  “Electromagnetic  Radiation  (EMR)-Matter  Interactions  (Remote  Sensing).” 

http://what-when-how.com/remote-sensing-from-air-and-space/electromagnetic-radiation 

-emr-matter-interactions-remote-sensing/ . 

The  distance  that  EM  radiation  can  penetrate  the  atmosphere  is  determined  by  the 
wavelength.  The  study  of  incident  radiation  absorption  is  of  special  interest  as  it  indicates 
the  wavelength  windows  of  radiation  that  can  be  detected  by  the  appropriate  sensors.  The 
atmospheric  windows  from  the  MWIR  of  3  to  5  micrometers  (pm)  to  the  LWIR  of  8  to 
12  pm,  where  transmissivity  is  higher,  are  the  ones  exploited  by  most  of  the  IR  sensors 
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(see  Figure  21).  Different  sensors,  which  operate  at  distinct  regions  of  the  EM  spectrum, 
generate  signals  based  on  the  sensing  mechanisms  employed. 


Figure  21.  Atmospheric  Transmission 
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Diagram  of  atmospheric  transmission  as  a  function  of  the  wavelength  Source:  “Remote 
Sensing,”  http://earthobservatory.nasa.gov/Features/RemoteSensing/remote  04.php. 


3.  Terms  and  Theory 

In  this  section,  some  basic  definitions  and  figures-of-merit  are  described  to  offer 
insight  into  the  expected  performance  of  the  sensors  used  in  this  thesis  research. 

•  The  field-of-view  (FOV)  is  the  angle  that  can  be  viewed  through  an 
aperture. 

•  The  instantaneous  field-of-view  (IFOV)  is  the  angle  that  corresponds  to 
each  pixel  of  an  array  and  measures  the  angular  resolution  of  the  detector. 

•  The  noise-equivalent  temperature  difference  (NETD)  is  a  measure  of  the 
image  noise  level  of  an  IR  camera  and  represents  the  temperature  change 
that  results  in  a  signal  equal  to  the  total  root-mean-squared  (RMS)  noise. 

•  A  blackbody  is  a  totally  non-reflective  object  that  does  not  transmit 
incident  radiation.  Instead,  it  absorbs  any  incident  radiation  that  strikes  it 
at  any  wavelength.  It  turns  out  that  good  absorbers  are  also  good  emitters. 
According  to  Wien’s  displacement  law  [43],  the  maximum  energy  is 
observed  at  a  wavelength  that  is  inversely  proportional  to  the  temperature 
of  the  source  blackbody.  This  statement  can  also  be  observed  in  the 
following  equation  and  Figure  22: 

 2.898x10  3  [777  W] 


•  Emissivity  is  defined  as  the  fraction  e  of  the  amount  of  radiation  emitted 
from  an  object  compared  to  that  emitted  from  a  blackbody  at  a  given 
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temperature.  It  is  given  by  a  number  between  0  and  1,  in  which  1  is  the 
emissivity  of  a  blackbody.  The  various  values  of  s  for  different  bodies  are 
given  in  tables,  and  sensors  have  to  be  adjusted  accordingly  before  any 
measurements  are  recorded. 

•  Emittance  is  defined  as  the  amount  of  energy  emitted  from  an  object  per 
unit  of  time  and  per  unit  of  area  (W/m2). 


Figure  22.  Blackbody  Spectra  for  a  Set  of  Temperatures 


X  Uim] 

This  graph  illustrates  blackbody’s  spectral  radiant  emittance  according  to  Wien’s 
displacement  law  as  a  function  of  wavelength  X  plotted  for  different  characteristic 
absolute  temperatures.  The  relevant  curves  for  non-blackbodies  (gray  bodies)  would  be  of 
the  same  shape  but  of  a  different  value  according  to  e.  From  this  graph  it  becomes 
obvious  that  missiles  will  be  most  evident  in  3-5  pm  window,  while  humans  in  8-12pm. 
Source:  http://how-it-looks.blogspot.com/2QlQ/01/infrared-radiation-black-bodies-and. 

html. 


4.  Imaging  Sensors 

The  imaging  sensors  fall  under  two  main  categories:  thermal  sensors  and  photon 
sensors,  or  photodetectors.  All  the  detectors  under  these  categories  respond  to  absorbed 
radiation. 
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a. 


Photodetectors 


Photodetectors  are  further  classified  into  photoconductors  and  photodiodes. 
Photoconductors,  used  in  IR  sensors,  are  semiconductor  devices  designed  to  detect 
radiated  photons,  the  fundamental  unit  of  EM  waves.  When  semiconductors  absorb 
photons,  electrons  in  the  valence  band  get  excited  and  move  to  the  conduction  band  [44], 
[45].  A  signature  is  formed  when  an  object  reflects  solar  radiance  or  the  object  itself 
emits  radiance.  Figure  23  shows  how  different  materials  respond  to  wavelengths  in  the  IR 
and  visible  spectra.  These  materials,  based  on  their  spectral  characteristics,  are  used  in  the 
manufacturing  of  focal  plane  arrays 


Figure  23.  Materials  Used  in  Photodetectors 
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Source:  R.  Olsen  C.,  “Remote  sensing  from  air  and  space,”  SPIE,  2007. 


b.  Thermal  Sensors 

Thermal  sensors  operate  based  on  temperature  changes  as  IR  radiation  is 
absorbed.  The  temperature  changes  are  subsequently  converted  into  electric  signals. 
Thermal  sensors  are  further  classified  into  thermoelectric,  pyroelectric,  and  bolometric 
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sensors.  In  bolometers,  which  are  thermally  isolated  resistors,  a  change  in  the  temperature 
of  the  sensor  will  affect  its  resistance.  The  temperature  coefficient-of-resistance  (TCR)  is 
used  to  quantify  the  resistance  changes  [41].  TCR  is  given  in  the  following  equation: 

1  A R 

a  = - 

R  AT 

in  which  AR  represents  the  change  of  the  bolometer  resistance  triggered  by  the 
temperature  variation  AT,  and  R  is  the  bolometer’s  resistance. 

c.  Focal  Plane  Arrays 

In  focal  plane  arrays  (FPAs),  the  flux  that  enters  the  lens  is  focused  on  the  plane 
where  the  array  of  the  sensors  is  located.  For  the  uncooled  infrared  imaging  sensors,  a  2D 
array  of  microbolometer  pixels  with  defined  spacing,  referred  to  as  the  pitch,  forms  the 
thermal  detector  (see  Figure  24).  The  changes  in  the  resistance  of  these  pixels  provide  the 
necessary  information  to  produce  an  image  from  this  thermal  camera. 


Figure  24.  Bolometer  Pixel 


In  the  schematic  drawing  of  the  bolometer  pixel,  some  key  materials  and  parts  are 
indicated.  Source:  F.  Forsberg,  A.  C.  Fischer,  N.  Roxhed,  B.  Samel,  P.  Ericsson,  G. 
Stemme,  and  F.  Niklaus,  “Heterogeneous  3D  integration  of  17  pm  pitch  Si/SiGe  quantum 
well  bolometer  arrays  for  infrared  imaging  systems,”  J.  Micromechanics 
Microengineering,  vol.  23,  no.  4,  p.  045017,  2013. 


In  visible  image  sensors,  the  two  main  categories  for  array  composition  include 
charge-coupled  devices  (CCDs)  and  complementary  metal-oxide -semiconductors 
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(CMOSes).  One  of  the  patterns  for  generating  color  images  is  shown  in  Figure  25.  In  a 
CCD,  photons  accumulate  on  the  sensors  while  in  a  CMOS,  a  photocurrent  attracts 
additional  electrons  to  the  sensor  [46],  [44].  The  steps  involved  for  the  final  image  by 
sensor  type  are  shown  in  Figure  26. 


Figure  25.  Bayern  Pattern  Filtration 
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Traditional  CCD  /  CMOS  sensor.  Source:  “BayerPatternFiltration,” 

https://commons.wikimedia.Org/wiki/File:BaverPattemFiltration.png. 


Figure  26.  Image  Sensing  Steps 
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Source:  R.  Szeliski,  Computer  Vision :  Algorithms  and  Applications,  vol.  5.  Springer, 
2010. 
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5. 


Sensors  Used 


Two  different  cameras,  a  FLIR  C2  and  a  FLIR  SC640,  were  used  for  collecting 
images,  both  hosting  dual  sensors  in  the  IR  and  visible  spectra.  Some  of  their  main 
characteristics  are  presented  in  Tables  4  and  5,  respectively.  The  IR  sensors  in  both 
cameras  comprise  uncooled  microbolometers  in  the  LWIR  region  of  the  IR  spectrum. 
The  big  difference  comes  from  the  IFOV  of  the  cameras,  in  which  we  see  a  significant 
difference  from  0.33  rnrad  to  11  mrads  and,  thus,  a  big  difference  in  the  spatial 
resolution.  Additionally,  in  the  thermal  sensitivity  of  the  cameras,  there  was  a  big 
difference,  from  30  to  100  milli-Kelvins  (mK).  In  spatial  resolution  and  in  thermal 
sensitivity,  FLIR  SC  640  demonstrated  better  characteristics. 


Table  4.  Technical  Characteristics  of  FLIR  C2  Camera 


Imaging  and  optical  data 

IR  resolution 

80  x  60  pixels 

NETD 

100  mK 

Field  ot  view 

45“ x  34° 

Minimum  locus  distance 

•  Thermal:  0.15  m  (0.49  ft.) 

•  MSX:  0.3  m  (1  ft.) 

Focal  length 

1.54  mm  (0.061  in.) 

Spatial  resolution  (IFOV) 

1 1  rnrad 

F-number 

1.1 

Image  frequency 

9  Hz 

Focus 

Focus  free 

Detector  data 

Focal  Plane  Array 

Uncooled  microbolometer 

Spectral  range 

8-14  pm 

Detector  pitch 

17  pm 

IR  sensor  size 

80x60 

Digital  camera 

Digital  camera 

640  x  480  pixels 

Digital  camera,  focus 

Fixed  focus 

Digital  camera,  FOV 

55°  x  42'  ±2" 

Source:  “User’s  manual  -  FLIR  Cx  series,”  http://www.farnell.com/datasheets/ 
1903471.pdf. 
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Table  5.  Technical  Characteristics  of  FLIR  SC640  Camera 


Imaging  and  optical  data 

IR  resolution 

640  x  480  pixels 

Thermal  sensitivity/NETD 

<30  mK  ©  +30°C  (+86°F) 

Field  of  view  (FOV)  /  Minimum  focus  distance 

12°  x  9“  / 1.2  m  (3.9  ft.) 

Spatial  resolution  (IFOV) 

0.33  mrad 

Image  frequency 

30  Hz  (60/1 20  Hz  with  windowing) 

Focus 

Automatic  or  manual  (electric  or  on  the  lens) 

Zoom 

1-8x  continuous,  digital  zoom,  including  panning 

Focal  Plane  Array  (FPA)  /  Spectral  range 

Uncooled  microbolometer  /  7.5-13  pm 

Digital  camera 

Built-in  digital  camera 

3.2  Mpixel,  auto  focus,  and  video  lamp 

Source:  “Technical  Data  -  FLIR  SC640,”  http://support.flir.com/DsDownload/ 
Assets/40402-3601  en  40.pdf. 


6.  Digital  Image  Representation  and  Data  Visualization 

Commission  Internationale  d’  Eclairage  (CIE)  has  standardized  RGB  image 
representation  using  red,  green,  and  blue  primary  colors,  as  shown  in  Figure  27.  Under 
this  standardization,  colors  are  characterized  by  a  point  described  by  the  x  and  y 
coordinates  and  luminance  Y,  which  is  a  measure  of  intensity  [47]. 


Figure  27.  The  CIE  Chromaticity  Diagram 


All  colors  visible  to 
the  average  human  eye 
:  are  contained  inside 
the  diagram 

520 


The  colors  along  any 
line  between  two 
•  points  can  be  made  by  : 
mixing  the  colors  at 
the  end  points.  In  this 
case  Green  +  Red  = 
Yellow 
<340 'f 


V 


The  edge  of  the  diagram, 
called  the  spectral  locus, 
represents  pure, 
monochromatic  light 
measured  by  wavelength  in 
nanometers.These  are  the 
most  saturated  colors. 


:  The  least  saturated  colors  are  at 
:the  center,  emanating  from  white. : 


0.0 

0.0  0.1  0.2 


•  Color  gamut:  subset  of 

colors  that  can  be 

•  represented  by  mixing 

the  colors  at  it's 


"line  of  purples":  these 
;  colors  are  fully  saturated 
but  can  only  be  made  by 
*  mixing  two  colors  (red  and 
blue) 


0.3  0.4  0.5  0.6  0.7  0.8 


Source:  “Anatomy  of  a  CIE  Chromaticity  Diagram,”  http://dot-color.com/ 
categorv/terminology/. 
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The  data  visualization  coming  from  an  image  is  a  grid  of  numbers  that  represents 
intensities  ranging  from  0  to  255.  In  true  color  images,  each  of  the  pixels  will  have  a  set 
of  three  values  that  directly  associates  with  the  red,  green,  and  blue  channels.  As  a  result, 
each  true  color  image  will  be  represented  by  a  three-dimensional  array  with  the  size  of 
each  being  the  same  as  the  size  of  the  image’s  pixels.  In  grayscale  images,  only  one 
matrix,  which  represents  only  brightness  information,  will  be  fonned  out  of  the  3D  array 
[48].  In  both  cases,  0  represents  black  and  255  white,  as  shown  in  Figure  28.  This  grid  is 
the  computer’s  view  of  an  image.  These  are  the  values  that  various  image -processing 
operators  use  as  inputs  to  produce  desired  patterns  or  as  outputs  for  further  analysis  and 
modeling  [46]. 


Figure  28.  Digital  Image  Data  Visualization 


Digital  image  is  composed  of  a  rectangular  pixel  array  in  (x,  y)  coordinate  with 
corresponding  intensity  values. 


C.  CLASSIFICATION  PROCESS 

The  classification  problem  belongs  to  the  category  of  supervised  learning 

problems,  in  which  we  attempt  to  map  inputs  for  specific  outputs  that  represent  our 

classes.  In  general,  classification  can  be  roughly  described  as  the  process  by  which 

models  created  with  past  data  can  be  used  to  predict  a  class  of  new  data  with  similar 
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attributes.  In  this  way,  discriminants  are  formed  as  classification  rules  that  can  distinguish 
between  the  examples  in  the  different  classes  [49].  The  following  steps  were  carried  out 
during  the  procedure  to  form  these  discriminants  in  our  binary  classification  problem. 

1.  Image  Processing 

Image  processing  is  the  process  through  which  captured  images  can  be  improved 
by  enhancing  characteristics,  such  as  brightness  and  contrast,  or  by  removing  the  noise 
[46],  [50].  This  step  brings  the  images  to  a  state  that  is  more  suitable  and  assists  the 
specific  feature-extraction  process. 

2.  Feature  Detection  and  Description 

Feature  detection  or  extraction  is  the  process  by  which  images  are  scanned  in 
order  to  extract  characteristic  locations,  or  subsets  of  the  image  space.  Consequently,  the 
regions  surrounding  these  locations  will  be  exploited  to  generate  invariant  descriptors  for 
our  feature-description  process.  These  descriptors  characterize  each  image  and  the  set  of 
images  for  each  class.  The  method  for  extracting  these  descriptors,  and  specifically, 
speeded-up  robust  features  (SURF),  is  analyzed  in  the  following  paragraphs.  This  is  a 
very  important  step  as  these  features  will  become  the  inputs  to  form  the  discriminants  for 
training  the  neural  network. 

a.  Speeded-Up  Robust  Features  (SURF) 

Since  the  targets  and  platfonns  in  battlefields  are  kinetic  in  unpredictable  ways, 
we  are  also  interested  in  the  timeliness  of  the  information.  SURF  features  were  finally 
selected  to  provide  us  the  necessary  descriptors.  These  features  operate  only  in  grayscale 
images,  are  rotation  and  scale  invariant,  and  mainly  used  for  matching  applications.  Other 
applications  in  which  these  features  can  be  used  include  image  registration,  3D 
reconstruction,  and  object  recognition,  as  indicated  by  inventors  [23].  Additionally,  these 
features  are  rapidly  computed,  extracted,  and  compared.  In  several  cases,  SURF  features 
were  used  to  solve  classification  problems  for  facial  expression  [51],  objects  and  their 
localization  [52],  automatic  image  annotation  [53],  and  image -matching  problems  using 
supervised  learning  [54]. 
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The  implementation  of  the  original  method  involves  three  steps  mainly  for 
matching  applications,  that  is  correspondence  between  images  [23].  As  a  first  step, 
appropriate  interest  points  should  be  detected  followed  by  the  description  of  the  area 
around  these  points  in  the  fonn  of  a  vector.  Candidates  for  the  interest  points  are  those 
that  represent  objects  in  the  image  such  as  blobs,  corners,  and  T-junctions.  The  metric  for 
this  step  is  a  Hessian  matrix-based  detector,  called  the  “Fast-Hessian”  detector,  that  relies 
on  integral  images  [55]  for  image  convolutions.  The  descriptors  have  64  dimensions  and 
describe  Haar-wavelet  responses  in  the  area  surrounding  the  interest  points. 


(1)  Fast-Hessian  Detector 

The  detector  of  the  interest  points,  based  on  the  Hessian  matrix,  is  characterized 
by  stability  and  repeatability.  Instead  of  a  Gaussian  or  a  Laplacian  of  Gaussian  filter,  an 
approximation  of  them  with  a  box  filter  is  used,  as  shown  in  Figure  29  (a).  In  this  way,  a 
scale-space  is  created,  as  shown  in  Figure  29  (d).  Because  we  use  integral  images,  there  is 
no  need  to  down-scale  the  images;  instead,  the  increased  dimensions  of  filters  are  applied 
to  the  integral  images,  as  shown  in  Figure  29  (b).  The  result  is  faster  perfonnance  by 
roughly  a  factor  of  three,  without  a  negative  effect  on  the  detection  of  the  interest  points. 
The  points  are  finally  localized  after  a  non-maximal  suppression  in  a  three  by  three  by 
three  neighborhood  [23],  [56],  [57],  as  shown  in  Figure  29  (c).  As  a  result,  the  detector  is 
based  on  the  detenninant  of  the  Hessian  matrix  for  both  the  scale  and  the  location  of  the 
interest  points.  Therefore,  the  Hessian  matrix  for  a  specific  point 
Pi=  (x,y)  on  image  Ixy  is  H  (p;,  a)  at  p;  at  a  scale  of  a,  defined  as  follows: 


LJPn°)  LJPi’&) 
KiPi’V)  Lyy(Pi^) 


in  which  Lxx(pi,c)  is  the  convolution  of  the  second  order  Gaussian  derivative  of  the  image 
Ixyat  point  pi  [56], 


44 


Figure  29.  SURF  Detector 


(c)  (d) 

In  (a):  LoG  and  box  filter  approximations.  In  (b):  the  way  the  box  filters  grow  while 
being  kept  to  scale.  In  (c):  the  non-maximal  suppression  that  results  in  the  detection  of 
the  interest  point.  In  (d):  the  scale-space  pyramids  for  LoG  (left)  and  box  filters  (right) 
Adapted  from  H.  Bay,  A.  Ess,  T.  Tuytelaars,  and  L.  Van  Gool,  “Speeded-Up  Robust 
Features  (SURF),”  no.  September,  2008  and  S.  Ili,  “Feature  Descriptors,”  Track.  Detect. 
Comput.  Vis.,  2010. 


(2)  Descriptors 

The  method  to  extract  the  descriptors  from  the  surrounding  area  of  the  interest 
points  includes  two  more  distinct  steps.  First,  the  Haar  wavelet  [50]  responses  will  have 
to  be  calculated  both  in  x  and  v  directions  within  a  certain  area  defined  by  scale 
parameters  to  detect  the  feature  direction.  In  the  second  step,  a  square  region  needs  to  be 
defined  around  the  interest  point  and  subsequently  rotated  to  match  the  feature  direction, 
as  shown  in  Figure  30  (a).  Sub-regions  of  four  by  four  are  further  defined,  wherein 
simple  features  are  computed  in  five  by  five  grids  inside  these  sub-regions,  as  shown  in 
Figure  30  (b).  The  final  64  elements  of  the  descriptor  compose  a  four-element  vector 
(Edx,  Z|dx|,  Xdy,  X|dy|)  multiplied  by  the  sixteen  sub-regions. 
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Figure  30.  SURF  Descriptor 
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In  (a):  the  scale-based  parameters  to  detect  the  feature  direction.  In  (b):  the  sub-regions, 
wherein  descriptor  elements  are  calculated.  Adapted  from  D.  Schmitt  and  N.  McCoy, 
“Object  Classification  and  Localization  Using  SURF  Descriptors,”  Object  Classif. 
Localization  Using  SURF  Descriptors,  pp.  1-5,  2011,  and  H.  M.  Sergieh,  E.  Egyed- 
Zsigmond,  M.  Doller,  D.  Coquil,  J.-M.  Pinon,  and  H.  Kosch,  “Improving  SURF  Image 
Matching  Using  Supervised  Learning,”  2012  Eighth  Int.  Conf.  Signal  Image  Techno  I. 
Internet  Based  Syst.,  pp.  230-237,  2012 


3.  Multi-Sensor  Data  Fusion 

If  we  consider  the  most  accurate  definitions  that  exist  concerning  data  fusion,  we 
can  define  multi-sensor  data  fusion  as  the  collection  and  combined  use  of  data  originating 
from  various  sources  to  acquire  enhanced  information  [58],  [59].  Data  fusion  techniques 
and  practices  have  been  increasingly  used  in  cases  where  a  variety  of  sensors  and  sources 
are  available.  When  such  techniques  are  implemented,  the  expectation  is  that  the  results 
will  become  more  accurate  and  reliable  than  data  acquired  from  one  source.  Furthermore, 
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information  fusion,  which  refers  to  the  fusion  of  already  processed  data,  can  be 
implemented  at  a  later  stage  or  in  combination  with  data  fusion. 

Several  classification  schemes  of  data  and  infonnation  fusion  have  been 
proposed,  mainly  because  it  is  a  technique  that  numerous  disciplines  can  implement. 
Castanedo  in  his  review  of  the  existing  data  fusion  techniques  [58]  presents  and 
categorizes  them  according  to  certain  criteria.  We  will  use  the  classification  based  on 
abstraction  levels  to  better  describe  the  different  ways  that  data  fusion  techniques  have 
been  implemented  in  our  experiment.  According  to  Castanedo’ s  classification  criterion, 
we  can  describe  the  following  categories: 

(1)  Low  level  fusion,  in  which  raw  data  are  fed  as  an  input  to  the  data  fusion 
process; 

(2)  Medium  level  fusion,  also  known  as  feature  or  characteristic  level,  in 
which  features  or  characteristics  are  used  as  inputs  to  the  fusion  process; 

(3)  High  level  fusion,  which  is  also  known  as  decision  fusion;  and 

(4)  Multiple  level  fusion,  in  which  data  fusion  involves  some  or  all  of  the 
previous  mentioned  categories  [58]. 

4.  Machine  Learning 

Although  machine  learning  uses  data  paradigms  to  form  or  predict  patterns  of 
data,  it  is  not  limited  in  finding  solutions  for  database  challenges.  The  main  focus  of  this 
science  is  to  produce  the  appropriate  models  that  will  enable  computers  to  demonstrate  a 
self-learning  performance  based  on  experience  built  on  past  data  without  being  explicitly 
programmed  [49],  [60].  Artificial  intelligence  (AI)  is  a  big  part  of  this  field.  Furthennore, 
artificial  neural  networks  (ANNs)  are  an  attempt  to  create  machine  intelligence  that  can 
mimic  human  brain  performance  in  the  areas  that  still  need  development,  such  as 
computer  vision  or  pattern  recognition.  In  this  thesis,  ANNs  are  used  to  build  the 
classifier  for  a  binary  classification  problem  since  they  are  proven  excellent  classifiers 
when  dealing  with  non-linear  problems. 
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a.  Artificial  Neural  Networks 

Neurons  are  the  basic  elements  of  the  human  brain.  Their  operation  is  to 
accumulate  input  signals  and  when  a  certain  threshold  is  reached,  they  will  trigger  an 
output  signal,  as  shown  in  Figure  31.  The  same  concept  is  the  basis  of  the  ANNs,  or  more 
simply,  neural  networks  (NNs).  A  network  of  artificial  neurons  can  in  most  cases  be 
modeled  to  simulate  the  structure  of  biological  neurons  [61]. 


Figure  3 1 .  Neuron  Schematic 
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The  input  signals  may  be  outputs  from  other  neurons.  When  sufficient  signals  are 
accumulated,  the  neuron  will  trigger  and  fire  an  output  signal.  Source:  S.  Raschka, 
“Artificial  Neurons  and  Single-Layer  Neural  Networks,”  2015,  http://sebastianraschka. 
com/Articles/2015  single  layer  neiirons.html. 


The  equivalent  design  of  the  artificial  neuron,  which  is  the  basic  processing 
element  in  NNs,  is  shown  in  Figure  32.  A  number  of  inputs  X;  is  assigned  to  coefficients 
or  weights  wy,  which  control  the  impact  of  the  specific  connection,  plus  a  bias  wo,  to 
make  the  model  more  general.  Each  perceptron  with  the  weighted  sum  of  the  inputs 
multiplied  by  the  weights  defines  a  hyper-plane. 
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Figure  32.  Artificial  Neuron 
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The  weighted  sum  of  the  inputs  multiplied  by  the  weights  plus  the  bias  is  fed  into  the 
activation  function  in  which  a  threshold  determines  whether  it  will  fire  an  output.  Source: 
M.  Triki,  H.  Chabchoub,  and  W.  Hachicha,  “A  neural  network-based  simulation 
metamodel  for  a  process  parameters  optimization:  A  case  study,”  Logistiqua,  2011  4th 
Int.  Conf.,  pp.  323-328,  2011. 


Neurons  are  grouped  in  layers,  which  are  fully  connected  through  communication 
channels  called  connections.  The  first  layer  is  the  input  layer  in  which  the  extracted 
outputs  from  the  previous  steps  will  be  entered  as  inputs.  Thus,  the  nodes  on  the  input 
layer  are  limited  by  the  number  of  the  features  that  we  define.  Next,  we  find  the  hidden 
layers,  which  are  the  layers  in  between  the  input  and  the  output  layers.  Although  there  are 
no  rules  to  detennine  the  number  of  hidden  or  total  layers  and  their  respective  nodes, 
some  empirical  nonns  do  exist  based  on  experimentation  and  previous  results  from 
various  categories  of  problems.  One  norm  is  that  usually  three  layers  are  enough,  but 
additional  layers  can  be  added  if  results  are  not  satisfactory.  The  input  layer  is  not  taken 
into  account  when  referring  to  the  number  of  layers  of  a  NN.  The  last  layer  is  the  output 
layer,  which  in  our  case  is  going  to  have  two  nodes  corresponding  to  the  two  classes  of 
our  problem,  as  shown  in  Figure  33. 
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Figure  33.  Connections  and  Layers  of  ANNs 


An  example  of  an  ANN  topology.  Source:  “Neural  Network  Classifier,”  https://code. 
google. co  m/p/nlkwiki/ClassifierNeuralNetwork. 


The  two  main  network  topologies  are  feedforward  and  feedback,  or  recurrent, 
neural  networks.  In  the  first  category,  neurons  do  not  back  propagate  while  in  the  latter, 
they  do.  One  more  difference  that  NNs  have  is  determined  by  the  kind  of  data  that  they 
can  handle.  The  main  data  categories  or  variables  that  exist  are  the  categorical  and 
quantitative  variables.  Categorical  data  needs  to  be  mapped  into  numbers  before  they  are 
fed  into  the  network  [61].  ANNs  can  be  characterized  with  a  set  of  numbers  that 
describes  their  topology.  For  example  an  X- 100-2  ANN  is  a  two-layer  ANN  with  X 
inputs,  100  neurons  in  the  hidden  layer  and  two  outputs,  and  it  is  the  characteristic 
topology  of  the  network  used  in  this  thesis. 

ANNs  usually  have  one  or  more  hidden  layers  with  neurons.  The  activation 
function  applied  on  the  weighted  sums  of  the  neurons  can  be  either  linear  or  non-linear. 
The  main  functions  that  can  be  met  in  NNs  are  presented  in  Figure  34.  In  classification 
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problems,  as  they  are  highly  non-linear,  the  equivalent  activation  functions  are  used.  In 
this  classification  problem,  specifically  the  logsig(n)  is  used  in  the  output  layer  since  we 
want  to  constrain  the  output  between  0  and  1  for  the  binary  classification  problem  [62]. 


Figure  34.  Transfer  Functions  of  ANNs 


The  main  and  most  common  activation  or  transfer  functions  that  perform  the  decision  in 
each  neuron  are  depicted  in  this  image:  tan-sigmoid  (a),  log-sigmoid  (b)  and  linear  (c). 
Modified  versions  of  them  can  be  also  used  depending  on  the  type  of  the  ANN.  Source: 
“Analysis  of  Pesticide  Mixtures  Using  Intelligent  Biosensors,”  http://www.intechopen. 
com/books/intelligent-and-biosensors/ analvsis-o  f-pesticide-mixtures-using-intelligent- 

biosensors 


b.  Training  the  Neural  Network 

For  the  training  process  of  the  ANN,  an  important  parameter  that  we  have  to 
consider  is  the  type  of  the  ANN  and  the  training  algorithm  that  it  uses.  In  MATLAB, 
several  training  algorithms  can  be  implemented  in  the  ANN,  and  the  choice  depends  on 
factors  such  as  the  complexity  of  the  problem  or  the  amount  of  the  available  data.  In 
classification  problems,  scale  conjugate  gradient  (SCG),  a  backpropagation  algorithm 
(trainscg),  seems  to  outperform  the  rest  of  the  available  algorithms,  especially  when  large 
number  of  weights  exist  [63],  [64]. 

The  typical  work  flow  for  the  overall  training  process  after  selecting  the  training 
algorithm  has  the  following  steps  [65]: 

•  Collect  and  prepare  the  data 

•  Create  the  network 

•  Configure  the  network  parameters 


51 


•  Initialize  the  weights  and  biases 

•  Train  the  network  using  the  training  data 

•  Validate  the  network 

•  Use  the  network  to  predict  the  testing  data,  which  has  to  be  separated  from 
the  training  and  validation  steps. 

c.  Classification 

The  problem  of  classification,  or  in  this  case,  category  (class)  recognition  and 
image  classification,  can  be  seen  as  the  task  of  creating  general  models  or  representations 
of  attributes  from  a  collection  of  data  that  characterizes  a  specific  class  then  finding  any 
existing  correspondence  between  any  set  of  testing  data.  Today,  image  classification 
continues  to  be  one  of  the  unresolved  problems  in  the  area  of  computer  vision  [46] .  That 
is  not  to  say  solutions  do  not  exist,  but  they  are  partial  and  applicable  only  to  certain  areas 
under  constraints.  The  main  comparison  often  used  to  support  this  statement  is  that 
current  solutions  cannot  outperfonn  the  capability  of  a  child  to  recognize  and  classify 
objects.  Over  the  last  fifteen  years,  it  seems  that  new  interest  in  NNs  has  made  steps 
toward  solving  this  problem  [66].  In  this  thesis,  the  last  step  is  to  evaluate  the 
performance  of  NNs  in  a  binary  classification  problem  of  categorizing  boats  between  the 
classes  of  sailing  and  power  boats. 

The  perfonnance  measures  used  for  this  classification  problem  are  the  confusion 
matrix  and  the  receiver  operating  characteristics  (ROC)  curves  [49].  For  a  binary 
classification  problem  like  the  one  investigated  in  this  thesis,  the  confusion  matrix  will 
have  the  fonn  of  Table  6,  wherein  the  corresponding  definitions  are  presented.  Further 
relevant  definitions  of  perfonnance  measures  are  given  in  Table  7. 
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Table  6.  Confusion  Matrix  for  Binary  Classification  Problem 


Predicted  class 


True  class 

Positive 

Negative 

Total 

Positive  (p) 

TP:  true  positive 

FN:  false  negative 

P 

Negative  (n) 

FP:  false  positive 

TN:  tme  negative 

n 

Total 

Pi 

nl 

N 

Adapted  from  E.  Alpaydin,  “ Introduction  to  Machine  Learning,'1  Thrird  Edi.  MIT  Press, 
2014. 


Table  7.  Performance  Measures  for  Binary  Classification  Problem 


Name 

Formula 

Error 

(FP+TP)/N 

Accuracy 

(TP+TN)/N=1 -Error 

TP  rate 

TP/p 

FP  rate 

FP/n 

Precision 

TP/pl 

Recall  or  TP  rate 

TP/p 

Sensitivity  or  TP  rate 

TP/p 

Specificity 

TN/n=l-FP  rate 

Adapted  from  E.  Alpaydin,  “ Introduction  to  Machine  Learning,"  Thrird  Edi.  MIT  Press, 
2014. 


ROC  curves  give  us  a  graphical  representation  of  the  true  positive  rate  (TPR)  as  a 
function  of  the  false  positive  rate  (FPR).  An  example  of  ROC  curves  with  a  rough 
representation  based  on  the  expected  classifier  performances  is  shown  in  Figure  35.  As 
shown  in  the  depicted  curves,  the  more  they  shift  to  the  upper  left  comer,  the  better  the 
classifier  is  characterized.  We  can  also  measure  the  quality  of  the  classifier  with  the  area 
under  the  curve  (AUC),  in  which  1  is  the  value  corresponding  to  an  ideal  classifier  [49]. 
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Figure  35.  Expected  ROC  Curves 


A  rough  representation  of  the  expected  ROC  curves  according  to  the  performance  of  the 
used  model  is  given  for  comparison  later  with  the  derived  results  from  our  binary 
classification.  On  the  x-axis,  FPR  is  the  ratio  of  false  positives  over  all  negatives,  and  on 
the  y-axis,  TPR  is  the  ratio  of  true  positive  over  all  positives.  Source:  “ROC  curves,” 
http://www.unc.edu/courses/2Q10falPecoP563/001/docs/lectures/ 

lecture22a.htm. 


Another  way  to  evaluate  the  perfonnance  of  an  ANN  is  to  use  the  mean  squared 
error  (MSE)  and  the  cross-entropy  or  deviance  as  a  measure  of  fit  [67],  [68],  [69].  MSE  is 
the  output  to  be  compared  during  the  training  or  testing  phase  of  the  ANN 
implementation.  The  error  is  the  difference  between  the  target  value  representing  an 
assigned  class  of  the  supervised  process  and  the  output  value  that  was  predicted  from  the 
ANN.  The  MSE  and  the  cross-entropy  are  given  in  the  following  equations: 
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1  N 

mse  =—YJ(t,-yi)2, 

A  i= i 

c  =  -^-^(tixlogyi  +  (l-ti)xlog(l-yi)), 

A  i= i 

in  which  N  represents  the  total  number  of  samples,  f  the  assigned  target  values,  and  y,  the 
predicted  outputs. 

D.  USING  CENETIX  TNT  TESTBED  FOR  EXPERIMENTATION  WITH 
SENSORS 

The  Center  for  Network,  Innovation  and  Experimentation  (CENETIX)  at  the 
Naval  Postgraduate  School  (NPS)  is  focused  on  extensive  experimentation  of  self¬ 
organizing  tactical  networking  and  collaboration.  Since  it  was  founded  in  2004,  many 
experiments  have  been  conducted  with  a  focus  on  adaptive  wireless  and  mesh  networks 
to  allow  connectivity  and  collaboration  of  various  manned  and  unmanned  platfonns,  from 
UAVs  to  divers  and  satellites  [70].  While  the  primary  focus  was  on  maritime  interdiction 
operations  (MIO)  initially,  numerous  partners,  whose  cooperation  supported  several 
national  as  well  as  international-level  field  experiments,  turned  it  into  a  leading 
innovative  solution  for  the  modem  battlefields.  Tactical  Network  Topology  (TNT)  and 
MIO  test-beds  allows  for  cooperation  and  experimentation  with  universities,  military 
institutions  and  organizations  such  as  MIT,  NMIOTC,  and  LLNL.  One  of  the  latest 
innovative  solutions  proposed  by  CENETIX  was  for  network  divers;  a  real-time 
exchange  of  images  and  text  was  tested  during  the  latest  experiments  in  San  Francisco 
Bay.  Acquired  results  from  this  and  other  experiments  will  be  subject  for  discussion  in 
this  thesis.  In  Figures  36  and  37,  the  tactical  and  reachback  infrastructure  as  well  as  the 
available  subnets  under  TNT-MIO  experimentation  are  presented. 
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Figure  36.  CENETIX  Tactical  and  Reachback  Infrastructure 
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The  CENETIX  tactical  and  reachback  infrastructure  is  extended  through  mesh  networks 
to  incorporate  various  heterogeneous  nodes.  Source:  A.  Bordetsky,  “Maritime  Threat 
Countering  Networking  and  Collaboration  Testbed.”  2013. 


Figure  37.  Subnets  under  TNT -MIO  experimentation 
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The  following  subnets  are  currently  configured  to  support  TNT-MIO  experimentation: 

•  100  -  OFDM  backbone  subnet  192.168.100.xx 

•  112  -  ITT  Mesh  192.168.112.0/24 

•  113  -  WaveRelay  Mesh  192.168.113.0/24 

•  137  -  Internet  Connection  Sharing  (ICS) 

•  72  -  SF  Bay  Area  MIO  192.168.72.xx 

•  84  -  Site300  subnet  192.168.84.0/24 

•  85  - CNTX  Reserved  cluster  192.168.85.0/24 

•  86  -  SF  Bay  Area  MIO  cluster  192.168.86.0/24 

•  96  -  Portion  of  TNT  subnet  192.168.96.0/22  (mask  255.255.252.0) 

•  97  -  Portion  of  TNT  subnet  192.168.96.0/22  (mask  255.255.252.0) 

•  98  -  Portion  of  TNT  subnet  192.168.96.0/22  (mask  255.255.252.0) 

•  99  -  Portion  of  TNT  subnet  192.168.96.0/22  (mask  255.255.252.0) 


Source:  “TNT-MIO  Host  IP  Configuration,”  http://cenetix.nps.edu/SAl/ipconfigmain. 
asp. 
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III.  SIMULATION  MODELING  OF  ATR  THROUGH  A  MESH 
NETWORK  OF  IMAGING  SENSORS 

Assets  operating  in  an  area  of  interest  (AOI)  follow  certain  procedures  to  classify 
and,  if  possible,  identify  contacts  of  interest  (COIs).  This  process  becomes  even  more 
challenging  when  only  passive  sensors  are  used.  Additionally,  there  are  operational 
restrictions  for  how  closely  these  assets  can  approach  any  of  these  potential  COIs.  These 
restrictions  depend  on  the  type  and  the  capabilities  of  the  assets  as  well  as  their  overall 
mission.  Oftentimes,  the  presence  of  other  friendly  assets  in  the  area  that  could  contribute 
to  this  effort  remains  unexploited.  The  proposed  scheme  of  this  thesis  aims  to  detect  and 
connect  multiple  assets  through  a  mesh  network  and  to  facilitate  the  timely  dissemination 
of  critical  information  coming  from  their  sensors,  in  a  littoral  environment.  This  scheme 
requires  that  available  distributed  database  centers  be  part  of  the  network.  Additionally, 
automation  should  be  applied  in  certain  functions  and  processes.  In  this  way,  received 
data  will  be  exploited  for  evaluation  in  the  decision-making  process  before  their  value 
expires. 

In  this  proposed  scheme,  we  focus  on  a  specific  task  of  classifying  image  data  of 
potential  targets  coming  from  the  mesh  nodes.  Some  of  the  challenges  that  this  task  poses 
are  the  need  for  high  throughput  as  well  as  the  ability  to  exploit  imagery  data  coming 
from  dissimilar  sensors.  The  purpose  of  the  simulations  and  experiments  carried  out  to 
support  the  concept  of  this  scheme  is  twofold.  First,  this  thesis  evaluates  the  feasibility, 
coverage,  and  quality  of  service  (QoS)  of  the  networked  sensors  in  a  maritime  littoral 
environment.  Second,  this  thesis  evaluates  data  fusion  and  classification  of  images 
coming  from  dissimilar  sensors.  In  this  way,  we  allow  assets  that  need  to  be  undercover 
to  safely  operate  in  the  area  while  other  platforms  that  can  take  the  risk  of  proximity  to 
the  target  capture  images  from  shorter  distances. 

In  this  chapter,  certain  steps  of  the  overall  conceptual  scheme  (see  Figure  38)  are 
explained  in  more  detail.  In  the  first  part,  the  steps  and  parameters  to  simulate  and 
evaluate  a  mesh  network  in  a  littoral  environment  are  presented  to  show  how  existing 
terrestrial  and  satellite  networks  can  be  extended.  Additionally,  CENETIX’s  contribution 
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to  the  scope  of  this  thesis  is  described.  In  the  second  part,  image  capture,  fusion,  training, 
and  classification  methods  used  in  this  thesis  are  further  explained. 


Figure  38.  Proposed  Scheme 


Proposed  scheme  for  ATR  through  wireless  mesh  sensor  networks.  If  and  when  the 
classified  COIs  are  positively  identified,  they  become  part  of  the  image  database,  and  the 
system  has  to  be  retrained. 


A.  SIMULATION  AND  EVALUATION  OF  A  WIRELESS  MESH  NETWORK 

Initially  and  according  to  the  STK  and  Qualnet  simulation,  a  mesh  network 
should  be  established  with  a  quality  of  service  (QoS)  that  could  facilitate  the  timely 
dissemination  of  the  captured  images.  At  least  one  of  the  participating  assets  in  the 
network  should  be  a  distributed  data  center  with  the  functionality  of  a  network  operating 
center  (NOC),  capable  of  monitoring  the  status  of  the  networked  assets.  These  assets 
could  be  unmanned  aerial  vehicles  (UAVs),  unmanned  ground  vehicles  (UGVs),  ground 
stations,  ships,  air  assets,  satellites,  or  even  human  agents  with  or  without  imaging 
capabilities  acting  as  relays  to  extend  the  network  in  the  latter  case.  Figure  39  depicts  this 
concept  and  displays  the  various  assets  in  a  littoral  environment.  In  this  challenging 
environment  and  through  the  use  of  the  proposed  technologies,  the  geographical  obstacles 
are  not  a  problem  for  the  establishment  of  communications  between  the  assets  in  the  area. 
Instead,  the  land  masses  are  used  in  a  way  that  offers  an  advantage:  hosting  ground 
stations  act  as  gateways  and  nodes  for  the  interconnection  of  the  existing  subnets  of  the 
dynamic  mesh  network.  The  synthesis  of  these  networks  allows  for  dissemination  of  the 
images  captured  by  the  heterogeneous  imaging  sensors  and  their  exploitation  through  the 
proposed  scenario  in  STK. 
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Figure  39.  Terrestrial  and  Satellite  Network 


Extension  of  terrestrial  and  satellite  network  through  a  wireless  mesh  network  (WMN)  in 
a  littoral  environment  to  connect  assets  with  imaging  capabilities 


1.  STK  Simulation 

The  STK  simulation  is  focused  in  littoral  waters  and  a  limited  area  of  interest 
(AOI)  (see  Figure  40),  mainly  because  of  the  use  of  imagery  satellites  and  the  need  to 
investigate  the  revisit  time  for  a  specific  area.  The  objects  inserted  in  the  scenario  include 
ships,  UAVs,  ground  vehicles,  ground  stations,  as  well  as  imaging  and  communications 
relay  satellites. 
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Figure  40.  AOI  in  STK 


The  AOI  used  for  our  simulation  as  extracted  from  STK  2D  graphics. 

The  parameters  and  configurations  of  the  simulated  assets  are  presented  in  this 
section.  Ground  and  sea  tracks  as  well  as  flight  orbits  are  designed  for  the  several  assets 
to  demonstrate  mesh-networking  flexibility  in  the  progress  of  time,  as  shown  in  Figure 
41.  The  ability  to  replay  the  scenario  offers  insight  into  and  rough  measurements  of  the 
link  ranges  and  communications  success  in  the  physical  layer.  The  final  scenario  will  be 
saved,  along  with  images  and  videos  from  2D  and  3D  graphics  windows,  for  future 
evaluation  of  the  fifth  generation  Wave  Relay  radios. 
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Figure  4 1 .  Assets  Used  in  the  Simulation 


Assets  used  in  the  simulation  and  their  designed  tracks  as  extracted  from  STK  2D 
graphics. 


The  objects  are  imported  in  the  simulation  by  inserting  them  into  the  scenario  via 
the  “Insert  STK  objects”  window.  In  doing  so,  sensors,  antennas,  transmitters,  and 
receivers  are  attached  to  them.  Thereafter,  they  can  be  accessed  for  necessary 
configurations  from  the  “Object  Browser,”  an  instance  of  which  is  shown  in  Figure  42. 
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Figure  42.  Instance  of  the  Object  Browser 
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Instance  of  the  Object  Browser  where  all  the  objects  are  listed  and  can  be  accessed  for 
configuration. 


The  imaging  and  communications  relay  satellites  are  shown  in  Figure  43.  The 
integration  of  imaging  cube-satellites  into  the  mesh  network  is  included  in  this  scenario 
to  demonstrate  capabilities  that  already  exist  and  to  evaluate  their  potential  contribution 
in  intelligence,  surveillance,  and  reconnaissance  (ISR)  missions  in  littoral  waters.  These 
satellites  can  provide  imagery  information  during  operations  in  areas  where  no  assets  are 
available  or  when  other  reasons  prevent  the  presence  of  friendly  assets.  Since  we  are 
dealing  with  low  earth  orbits  (LEOs),  we  expect  the  revisit  times  in  the  specified  AOI  to 
be  small.  However,  if  we  can  monitor  the  timeframes  that  these  assets  are  available,  the 
gain  becomes  obvious. 
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Figure  43.  Satellites  in  Simulation 


The  satellites  and  their  low  earth  orbits  as  extracted  from  the  STK  scenario  3D  graphics. 


The  orbit  parameters  that  were  imported  into  STK  for  the  imaging  satellites  are 
presented  in  Figure  44.  The  height  was  set  at  300  km,  and  an  inclination  of  40  degrees 
was  assigned. 


Figure  44.  Orbit  Parameters  for  Imaging  Satellites 
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The  basic  orbit  parameters  that  were  assigned  for  the  three  imaging  satellites  were 
imported  in  the  Orbit  window  of  the  Basic  menu  of  the  object. 
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The  angle  at  which  the  sensor  is  pointing,  or  the  look  angle,  ,  was  set  at  nadir,  and 
a  ficld-of-rcgard  (FOR)  was  assigned  based  on  the  capabilities  of  the  sensor  to  alter  its 
look  angle.  FOR  sets  the  boundaries  where  the  field  of  view  (FOV) — the  sensor’s  cone  of 
visibility — can  move.  Figure  45  shows  the  FOR  and  FOV  that  were  simulated.  The 
concept  of  using  these  parameters  is  that  the  satellite  will  be  tasked  to  change  its  look 
angle  to  acquire  images  from  the  desired  area  based  on  operational  needs  during  the 
available  time  slots. 


Figure  45.  FOR  and  FOV  for  the  Imaging  Satellites 


The  FOR  and  FOV  projected  on  the  earth’s  surface  are  depicted  in  this  figure.  The  FOV 
can  be  assigned  a  look  angle  inside  the  FOR  boundaries  as  the  satellite  travels  in  its 
designated  orbit. 


Based  on  the  following  equation,  the  height  of  300  km  was  selected  to  provide  the 
desired  ground  sample  distance  (GSD)  or  resolution  calculated  for  a  nadir  of  3.66  m: 

GSD=2MxZxh, 

D 
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in  which  k  is  the  wavelength,  h  the  distance  between  the  sensor  and  the  object,  and  D  the 
sensor’s  aperture  diameter. 

The  UAV  used  for  the  simulation,  an  RQ-11B  Raven,  was  extracted  from  the 
STK  database.  Its  basic  parameters  are  shown  in  Figure  46.  A  flight  path  was  designed  in 
the  basic  route  menu  and  the  height  was  set  to  200  ft. 


Figure  46.  Basic  UAV  Parameters 


Propagator:  Mission  Modeler 
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The  rest  of  the  imported  nodes  in  the  simulation  were  communications  relay 
satellites,  ships,  ground  facilities,  and  ground  vehicles.  Routes  were  assigned  to  all  the 
kinetic  nodes  to  demonstrate  and  study  the  expected  fluid  nature  of  the  established  links. 
Finally,  the  Wave  Relay  radios  were  simulated  with  the  parameters  that  are  presented  in 
Figure  47.  These  parameters  are  not  the  precise  parameters  with  which  the  actual  radios 
may  be  configured.  Since  Wave  Relay  runs  proprietary  software,  these  settings  were  not 
open  and,  thus,  impossible  accurately  simulate.  Care  was  taken,  though,  for  the 
theoretical  performance  of  the  selected  settings  to  be  comparable  to  those  presented  in 
Chapter  II. 
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Figure  47.  Wave  Relay  Transmission  and  Antenna  Parameters 
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Some  of  the  parameters  that  were  inserted  for  the  configuration  of  the  Tx  and  antenna 
simulating  the  Wave  Relay  radios  are  shown  in  this  figure. 


2.  Simulation  in  QualNet 

For  the  QualNet  simulation,  the  nodes  participating  in  the  network  were 
configured  according  to  the  available  Wave  Relay  radio  parameters  presented  in  Table  1. 
A  second  separate  replay  was  possible  through  QualNet  that  gave  us  metrics  to  evaluate 
the  network  QoS  from  the  MAC  layer  to  the  application  layer  of  the  OSI  seven-layer 
model.  An  instance  of  the  QualNet  configuration  browser  and  settings  of  the  channel 
properties  window  are  shown  in  Figure  48.  As  previously  mentioned,  the  simulation 
could  not  be  entirely  accurate  since  the  exact  parameters  of  Wave  Relay  have  not  been 
released.  Therefore,  what  we  sought  from  this  simulation  was  to  set  the  basis  that  permits 
the  comparison  of  simulation  results  with  the  ones  obtained  in  future  field  experiments 
using  fifth  generation  Wave  Relay  radios.  Consequently,  after  a  fine-tuning  of  the 
parameters  in  the  simulation,  the  results  will  be  comparable  to  the  observed  results  in  the 
field.  Thus,  it  will  be  feasible  for  future  scenarios  and  experiments  to  be  simulated  and 
adjusted  accordingly  to  meet  the  desired  requirements. 
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Figure  48.  Qualnet  Nodes  Configuration  Browser 
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The_scenario  --  Scenario  Configuration  --  Channel  Properties 

Property  Value 


Number  of  Channels 


Channel  Name  [0] 

channel  1 

| .  Channel  Frequency  [0] 

2.4  GHz 

Enable  Inter-channel  Interfemce  [0] 

No 

Signal  Propagation  Speed  (m/s)  [0] 

300000000 

Propagation  Limit  (dBm)  [0] 

-1000 

\ .  Channel  Name  [1] 

channel  2 

Channel  Frequency  [1] 

2.4  GHz 

Enable  Inter-channel  Interfemce  [1) 

No 

Signal  Propagation  Speed  (m/s)  [1] 

300000000 

Propagation  Limit  (dBm)  [1] 

-1000 

Rain  Outage  Percent 

0.1 

The  participating  nodes  in  the  mesh  network  are  listed  in  the  configuration  browser. 
Channel  properties  that  were  imported  are  also  depicted. 


B.  IMAGE  FUSION  AND  CLASSIFICATION  SCHEME 

The  performed  research  in  this  thesis,  even  though  limited  due  to  lack  of 
equipment  and  resources,  proposed  a  scheme  for  classification  of  images  from  potential 
targets.  Additionally,  one  of  the  main  goals  was  to  evaluate  the  gains  from  applying  a 
fusion  step  for  the  images  coming  from  dissimilar  sensors  in  the  field.  This  final  proposal 
should  also  be  applicable  to  littoral  environments  that  pose  additional  challenges  in  terms 
of  the  wide  variety  of  backgrounds  that  the  acquired  images  usually  have. 

All  the  steps  that  were  finally  implemented  are  illustrated  in  Figure  49.  These 
steps  start  with  capturing  images  in  the  field  and  end  with  classifying  available  targets. 
The  methodology  used  at  each  of  these  steps  is  described  in  detail  to  show  the  challenges 
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that  were  posed  and  taken  under  consideration  through  the  overall  process.  A  MATLAB 
script  was  developed  for  this  implementation. 


Figure  49.  Image  Classification  Scheme 


Feature  Extraction 
and  Fusion  Step 


Training  Step 


Testing 

Step 


Proposed  scheme  for  the  image  classification  of  the  targets  that  includes  the  necessary 
steps  from  image  processing  to  the  final  step  of  image  classification. 


1.  Image  Capture  and  Collection 

Since  the  intention  of  this  research  was  to  study  all  the  concepts  under  the  littoral 
environment,  the  image  set  had  to  reflect  this  case.  Furthermore,  we  had  to  investigate  the 
impact  of  fusion  in  the  classification  process.  One  of  the  requirements  in  the  proposed 
scheme,  then,  was  to  handle  images  randomly  captured  with  no  control  over  their  aspects, 
distances,  background,  or  lighting  conditions.  Thus,  the  image  set  had  to  be  acquired 
from  the  field  since  no  images  were  available  that  included  the  same  scene  captured  by 
dissimilar  sensors.  Both  cameras  had  these  capabilities,  which  were  described  in  Chapter 
II.  The  final  image  set  included  three  different  categories  of  images,  as  shown  in  Figure 
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50.  All  categories  were  directly  acquired  by  the  cameras,  except  for  the  multispectral 
images  from  FLIR  SC640,  which  were  derived  after  an  image-to-image  registration 
process. 


Figure  50.  Categories  of  Captured  Images 
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The  image  set  included  color-visual  and  thermal  images  from  both  cameras  and 
additional  MSX  images  from  FLIR  C2.  The  multispectral  version  from  FLIR  SC  640  was 
derived  after  registration  of  the  color-visual  and  thermal  images. 


The  available  targets  in  the  area  were  powerboats  and  sailing  boats.  Hence,  the 
images  in  our  experiment  were  captured  to  include  these  targets  with  various 
backgrounds.  The  background  was  selected  to  be  either  the  sea  or  combined  sea  and  land 
to  simulate  the  background  of  images  expected  in  littoral  waters  (see  Figure  51). 
Background  extraction  of  the  images,  then,  became  a  very  complex  task,  which  was  not 
accomplished — as  different  algorithms  and  methods  used  were  able  to  extract  the 
background  from  only  part  of  the  image  set. 
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Figure  5 1 .  Sample  Images  with  Different  Background 


Images  were  purposely  captured  with  different  backgrounds  to  pose  the  challenge  of 
classifying  boats  in  such  a  complex  background. 


The  total  number  of  the  dataset  was  also  a  key  factor  for  performance  during  the 
whole  process.  The  total  number  of  images  included  in  the  experiment  was  85.  A  large 
number  of  images  were  rejected  because  of  the  poor  resolution,  especially  in  IR  for 
longer  distances  or  when  too  many  targets  were  in  the  image.  The  cameras  did  not  have 
advanced  capabilities  or  zoom  features  usually  met  in  cameras  used  for  ISR  purposes. 
Instead,  the  acquired  images  were  preprocessed  in  such  a  way  that  zoom  was  simulated, 
so  only  one  boat,  or  at  least  boats  only  from  the  same  category,  would  be  included  in  the 
image  (see  Figure  52). 


Figure  52.  Original  and  Zoomed  Images 


(a)  (b) 


Original  (a)  versus  zoomed  (b)  images  in  order  for  the  object  to  cover  the  majority  of  the 
image. 
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2.  Low  Level  Image  Fusion  Implementation 

Data  fusion  from  the  two  sensors  was  a  process  that  investigated  whether  images 
coming  from  different  sensors,  either  similar  or  dissimilar,  could  be  handled  by  one 
system  in  our  proposed  scheme  for  ATR.  Additionally,  we  examined  whether  fused 
images  would  result  in  better  classification  performance  outcomes.  Therefore,  the  fusion 
process  was  implemented  in  several  stages.  Initially,  after  the  images  were  captured,  low- 
level  fusion  had  to  be  applied  to  the  FLIR  DSC  640  thermal  and  visual  images  to  produce 
multispectral  ones.  For  this  implementation,  ENVI’s  image-to-image  registration  was 
finally  used — after  many  unsuccessful  trials  using  feature-based  registration  in 
MATLAB.  The  best  result  in  MATLAB  and  the  final  registered  image  for  one  pair  of 
images  are  shown  in  Figure  53.  The  steps  for  the  final  implementation  are  described  in 
the  appendix. 


Figure  53.  Fusion  Result  Comparison  from  MATLAB  and  ENVI 


(a)  (b) 

In  (a):  the  best  derived  result  after  feature-based  registration  in  MATLAB.  In  (b):  the 
final  registered  image  in  ENVI. 


Moreover,  images  from  different  sources  and  sensors  were  fused  in  sets  of  images 
falling  under  the  same  category,  in  our  case,  either  color-visual  or  multispectral  (MSX). 
The  thennal  images  from  FLIR  C2  demonstrated  very  poor  results  due  to  the  poor 
resolution,  so  they  were  not  fused  with  the  other  images.  The  category  of  MSX  images 
involved  images  already  fused  in  the  FLIR  C2  and  the  multispectral  products  of  image 
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registration  from  the  FLIR  SC640’s  pairs  of  images.  The  third  case  in  which  fusion  was 
implemented  was  the  medium-level  fusion  after  the  feature  extraction  step,  which  is 
discussed  later  this  chapter.  Finally,  all  the  images — MSX,  thermal,  and  visual — were 
fused  to  investigate  whether  the  classification  performance  was  improving. 

3.  Image  Processing 

In  the  image  processing  step,  all  images  were  converted  to  grayscale,  and 
subsequently,  histogram  equalization  was  applied  as  shown  in  Figure  54.  Initially,  the 
images  were  .jpg  files,  which  means  they  were  represented  in  MATLAB  as  three- 
dimensional  matrices  with  the  number  of  elements  equal  to  the  pixels  of  the  images,  as 
discussed  in  Chapter  II.  This  step  was  mandatory  since  the  feature-extraction  method 
works  only  with  grayscale  images,  which  were  represented  by  individual  matrices  of  the 
same  size  as  the  original  three  color  channels.  The  values  of  the  elements  of  these 
matrices  were  the  intensities  on  which  pixel  operations  were  performed. 

Figure  54.  Image  Processing 

Image  Processing  Step 


I 


L _ j 

Preprocessed  images  were  converted  to  grayscale,  and  histogram  equalization  was 
subsequently  applied. 

Applying  histogram  equalization  in  the  images  was  also  necessary  after  the  results 
were  derived  without  taking  this  step.  Because  of  the  limited  capabilities  to  adjust 
brightness  and  contrast  before  capturing  each  image,  the  extracted  SURF  features  were 
very  poor,  especially  in  the  lower  analysis  images.  Since  histogram  equalization  is  a 
gray-level  mapping  process  and  there  is  no  gain  when  applying  it  to  separate  color 
channels,  it  was  applied  after  the  step  for  changing  images  to  grayscale.  This  process 
calculated  the  probability  density  function  (PDF)  of  the  normalized  intensities  of  the 
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images’  pixels,  ranging  from  0  to  1.  Then,  cumulative  distribution  function  (CDF)  was 
used  as  a  transformation  function  [50].  The  result  of  the  histogram  equalization  was 
enhancement  and  uncovering  of  details  that  had  been  lost  due  to  poor  contrast  ratios 
[46],  [71];  an  example  is  given  in  Figure  55. 


Figure  55.  Image  before  and  after  Histogram  Equalization 


(a) 


(c)  (d) 

In  (a)  and  (b),  the  original  image  and  its  histogram  are  presented  while  in  (c)  and  (d),  the 
corresponding  results  after  histogram  equalization  was  applied. 


4.  SURF  Features:  Detection  and  Description 

The  next  step  was  the  extraction  of  the  features  to  be  used  as  inputs  in  the 
artificial  neural  network  (ANN).  Speeded-up  robust  features  (SURF)  were  finally 
selected  and  used  for  this  step  as  they  produce  scale  and  rotation  invariant  features.  These 
characteristics  are  particularly  important  in  the  maritime  environment  since  platforms  are 
constantly  moving  at  unpredictable  directions.  Additionally,  the  detection  of  the  interest 
points  and  the  extraction  of  the  corresponding  descriptors  were  fast  enough  to  support 
real  or  near  real-time  recognition  based  on  these  features.  The  steps  that  were  carried  out 
to  finalize  the  features  fed  into  the  ANN  are  presented  in  Figure  56. 
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Figure  56.  Steps  for  SURF  Features  Extraction 

SURF  features  extraction  steps 


After  image  processing,  SURF  features  were  extracted,  and  the  final  features  to  be  fed  in 
the  ANN  were  extracted  after  normalization  of  the  ten  strongest,  as  shown  in  the  step-by- 
step  process. 

When  SURF  features  were  extracted  for  the  image  set,  we  noticed  that  a  large 
number  of  images  were  fonned  around  interest  points  from  the  background,  thus  creating 
multiple  potential  outliers  that  would  eventually  degrade  the  classification  process.  This 
problem  could  have  been  avoided  if  background  extraction  had  been  possible  in  the 
previous  steps,  and  a  larger  number  of  the  strongest  features  selected  could  have  been 
possible  early  in  the  process.  After  experimentation  with  the  number  of  the  strongest 
features  in  each  image,  we  identified  the  ten  strongest.  This  selection  provided  enough 
features  from  various  characteristic  points  of  the  boats  to  significantly  reduce  the  outliers 
in  the  majority  of  the  images.  The  results  of  detecting  SURF  features  in  one  of  the  images 
are  presented  in  Figures  57  and  58.  As  shown  in  these  two  images,  the  SURF  features 
before  and  after  histogram-equalization  was  applied  differed  but,  still,  no  outliers  were 
detected. 
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Figure  57.  Total  and  Ten  Strongest  SURF  Features  in  an  Image 


(a)  (b) 

In  (a),  all  the  detected  SURF  features  are  displayed  while  in  (b),  only  the  ten  strongest. 
All  outliers  were  removed  in  this  image  by  this  reduction  of  selecting  only  the  ten 
strongest. 


Figure  58.  SURF  Features  after  Histogram  Equalization 


The  ten  strongest  features  detected  in  an  image  after  histogram  equalization.  Notice  that 
these  features  are  different  from  the  ones  detected  in  Figure  57  (b). 


The  final  step  taken  before  the  final  features  were  fonned  was  the  normalization 
step.  The  maximum  value  of  each  feature  vector  was  detected  and  used  to  normalize  the 
64  values  of  which  each  vector  was  composed.  With  this  step,  potential  matching  and 
scaled  patterns  of  feature  vectors  could  become  easier  for  the  ANN  to  associate. 
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5.  Medium  Level  Image  Fusion  Implementation 

During  this  step,  feature  vectors  derived  from  the  pairs  of  the  visual  and  the 
thermal  images  were  combined  to  investigate  whether  better  classification  performance 
occurred.  The  way  the  fusion  of  the  features  was  applied  is  depicted  in  Figure  59,. 


Figure  59.  Medium  Level  Fusion  of  SURF  Features 


The  extracted  SURF  features  from  the  thermal  and  visual  images  were  fused  to  create 
another  feed  for  the  ANN. 


6.  Image  Classification  Using  an  Artificial  Neural  Network 

The  classification  process  involves  training  an  ANN,  extracting  the  net  with  its 
weights,  and  finally,  using  it  to  classify  the  samples  of  the  testing  data,  which  were 
separated  to  ensure  that  they  were  not  involved  in  the  previous  steps.  In  the  following 
subsections,  the  procedure  that  was  followed  in  these  steps  is  further  explained.  Figure  60 
presents  the  processes  during  the  classification  step. 
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Figure  60.  Steps  for  Image  Classification  with  ANNs 


a.  Training  the  Artificial  Neural  Network 

Initially,  the  topology  of  the  ANN  had  to  be  defined.  As  we  had  to  deal  with  a 
binary  classification  process,  as  previously  discussed,  the  number  of  outputs  at  the  output 
layer  had  to  be  two.  The  number  of  neurons  in  the  input  layer  was  always  equal  to  the 
number  of  the  input  features  in  all  the  different  cases  we  tested;  these  sets  are  presented 
in  Table  8.  In  order  to  determine  the  final  number  of  neurons  in  the  hidden  layer,  several 
tests  were  conducted  that  ranged  from  20  to  400  neurons.  It  was  important  to  have 
enough  neurons  to  deal  with  the  complexity  of  the  problem.  However,  too  many  neurons 
could  result  in  overfitting  [72].  In  the  latter  case,  even  if  results  during  training  appeared 
to  be  ideal,  the  results  after  testing  would  not  be  at  an  acceptable  level.  As  a  result,  the 
final  number  of  neurons  at  the  hidden  layer  was  set  to  100,  for  which  the  conducted  tests 
presented  the  best  performance  for  the  classification  process.  Hence,  the  topology  finally 
selected  was  X- 100-2,  as  shown  in  Figure  61,  in  which  X  was  the  number  of  inputs  for 
each  of  the  tests. 
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Table  8.  Description  of  Tests  and  Number  of  Inputs 


Trial 

No 

Description 

No  of 
inputs 

No  of 
images 

1 

Visual  images  from  DSC640 

640 

62 

2 

Thermal  images  from  DSC640 

640 

62 

3 

MSX  images  from  DSC640 

640 

62 

4 

Fused  features  from  visual  and  thermal 
images  from  DSC640 

1280 

62 

5 

Mixed  visual  images  from  both  cameras 

640 

85 

6 

Mixed  MSX  images  from  both  cameras 

640 

85 

7 

MSX,  thermal,  and  visual  images  from 
both  cameras 

640 

232 

The  different  sets  coming  from  the  corresponding  sets  of  images  or  their  fusion  and  the 
fusion  of  features  that  are  tested  are  presented  in  this  table.  The  results  will  be  compared 
to  drive  our  conclusions. 


Figure  6 1 .  ANN  Topology  during  Trials 


This  ANN  topology  was  640-100-2.  There  were  640  inputs  for  the  input  layer,  100 
neurons  for  the  hidden  layer,  and  two  outputs. 

After  the  images  were  separated  in  training  and  testing,  a  target  class  was 
assigned  to  each  one  of  them  as  is  required  for  the  supervised  learning  problem  of 
classification  [49].  In  our  case,  we  had  a  binary  classification  problem,  so  the  binary 
classes  of  01  and  10  were  assigned  as  target  labels.  The  goal  for  the  ANN  in  the  training 
process  was  to  leam  to  map  the  inputs  to  the  predefined  outputs  or  targets. 

Additionally,  and  because  of  the  small  number  of  samples,  early  stopping  was 
applied  during  the  training  process  to  further  reduce  the  possibility  of  overfitting  [72], 
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[73].  In  order  to  implement  this  step,  during  each  ANN  training,  the  training  set  was 
further  split  into  training  and  validation  sets.  When  the  performance  of  the  validation  set 
during  training  began  to  increase  for  a  number  of  epochs  without  the  same  behavior 
detected  in  the  training  set,  the  training  stopped.  The  updated  weights  of  the  epoch  in 
which  the  overfitting  initiated  were  stored  for  the  ANN,  as  shown  in  Figure  62. 
Whenever  early  stopping  was  not  selected  for  the  trials,  the  classification  rate  during 
training  always  reached  100  percent,  but  the  testing  results  obtained  were  poor  and  rarely 
exceeded  40  percent,  which  was  a  clear  indication  of  overfitting. 


Figure  62.  Early  Stopping 

Best  Validation  Performance  is  0.6975  at  epoch  7 


During  the  training  process,  the  validation  performance  started  increasing  at  epoch  7 
while  the  same  did  not  happen  for  training  performance.  Training  stopped  at  epoch  13, 
but  the  weights  that  were  stored  were  from  epoch  7. 


b.  Classification 

The  final  step  was  to  obtain  the  classification  output  after  the  test  data  was  used  to 
evaluate  the  network.  The  techniques  implemented  in  this  case  to  compensate  for  the 
small  number  of  samples  were  to  retrain  multiple  times  and  to  retrain  multiple  networks 
since  training  sessions  started  with  different  initial  weights  and  biases  [72].  In  the  first 
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case,  a  network  was  trained  1000  times,  and  in  the  second,  ten  different  networks  were 
trained  100  times  each.  The  only  change  in  the  ten  NNs  occurred  in  the  different  training 
sets,  and  as  a  result,  testing  sets  as  bootstrap  methods  were  used.  The  basic  concept  of 
bootstrapping  was  to  randomly  select  training  datasets  with  replacements  from  the  total 
set  of  images  that  were  formed  in  each  case  [69].  In  both  cases,  1,000  values  of  the  mean- 
squared-error  (MSE),  cross-entropy,  and  classification  perfonnance  as  well  as  their 
averages  were  recorded  and  plotted.  These  averages  were  used  as  a  metric  to  evaluate  the 
performance  of  the  network.  The  ANN  that  yielded  the  lowest  MSE  or  cross-entropy  or 
the  highest  classification  rate  would  be  the  one  that  best  generalized  to  the  test  set. 
Finally,  the  derived  results  for  each  of  the  cases  of  Table  8  were  compared  to  help  us 
evaluate  whether  any  fusion  process  yielded  better  results  than  the  cases  in  which  only 
one  type  of  image  or  feature  was  used. 

C.  CENETIX  EXPERIMENTATION  MODELING 

CENETIX  field  experimentation  was  used  in  this  thesis  to  test  the  hypotheses  as 
well  as  support  and  enhance  the  conceptual  idea  of  the  proposed  modeling.  Several 
experiments  were  carried  out  during  the  recent  constellation  experiments  that  took  place 
in  San  Francisco  Bay  during  the  last  week  of  October.  The  results  of  the  experiments  this 
thesis  could  make  use  of  were  the  ones  involving  imagery  data  dissemination  from 
unmanned  or  manned  systems  as  well  as  the  decision  support  tools  that  were  tested  after 
the  appropriate  parameters  were  implemented  to  support  Wave  Relay  radio  monitoring 
[74].  Innovative  solutions  have  always  been  the  aim  for  CENETIX  experimentation.  Of 
particular  interest  for  this  thesis  were  the  following  three  aspects  of  the  CENETIX 
experiments:  exchanging  imagery  data  between  divers  and  an  unmanned  underwater 
vehicle  (UUV)  via  two-way  communications;  streaming  video  from  an  umnanned  ground 
vehicle  (UGV)  performing  a  land  search;  and  integrating  CodeMettle  network  monitoring 
and  management  tools. 

1.  Mesh  Networking  Implementations  for  Imagery  Data  Dissemination 

Mesh  networking  scenarios  were  planned  during  the  recent  CENETIX 
experimentation  and  collaboration  session  in  San  Francisco  Bay  that  included 

80 


disseminating  imagery  data  and  streaming  video.  Manned  systems,  unmanned  systems, 
and  divers  were  planned  as  nodes  that  would  extend  the  TNT-MIO  backbone.  Depending 
on  the  network’s  availability  Wave  Relay,  satellite  link,  and  4G  would  be  used.  Previous 
experiments  had  resulted  in  successful  networking  of  unmanned  systems  and  swimmers, 
as  shown  in  Figure  63.  A  step  forward  was  attempted  to  include  divers  among  the 
networked  assets. 


Figure  63.  Networked  Swimmers 


Swimmers  were  successfully  incorporated  in  the  network  in  previous  experiments. 


2.  CodeMettle’s  Network  Monitoring  and  Management  Tool 

CodeMettle,  as  it  is  open  source  software,  was  configured  to  monitor  Wave  Relay 
mesh-network  traffic  and  performance  [75].  This  was  feasible  since  Wave  Relay  could  be 
configured  and  monitored  through  the  supporting  application  programming  interface 
(API).  This  interface  could  run  over  the  Wave  Relay  hypertext  transfer  protocol  over  SSL 
or  secure  (HTTPS)  interface,  thus  allowing  communication  with  programs  that  submitted 
requests  conforming  to  the  hypertext  transfer  protocol  (HTTP)  [6],  [37].  The  data  and 
metrics  enabled  in  management  interface  could  be  monitored  through  the  Code  Mettle 
network  management  support  tool  when  configured  according  to  the  API.  As  a  result, 
signal-to-noise  ratio  (SNR),  traffic  load,  node  status,  GPS  information,  channel  usage, 
and  other  parameters  were  monitored  and  graphically  analyzed  to  enhance  SA  concerning 
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network  nodes  and  enable  end-to-end  network  management.  An  instance  of  CodeMettle 
management  tool  graphics  is  shown  in  Figure  64. 


Figure  64.  CodeMettle  Management  Tool  Graphics 
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IV.  SIMULATION  AND  EXPERIMENTATION  RESULTS 


The  steps  and  methods  applied  for  the  simulation  modeling  of  automatic  target 
recognition  (ATR)  through  a  mesh  network  of  imaging  sensors  described  in  the  previous 
chapters  gave  us  a  variety  of  results  that  are  presented  in  the  following  sections.  The 
results  are  separated  in  three  categories.  The  first  category  includes  the  results  obtained 
from  the  mesh  network  simulation  in  Systems  Tool  Kit  (STK)  and  QualNet.  Next,  the 
relevant  Center  for  Network,  Innovation  and  Experimentation  (CENETIX)  results  from 
field  experiments  are  presented.  The  third  and  final  category  involves  the  results 
concerning  the  image  fusion  and  classification  process. 

A.  STK  AND  QUALNET  SIMULATION 

Several  problems  were  detected  and  delayed  the  STK  and  QualNet  simulation  of 
the  mesh  network  of  multiple  heterogeneous  nodes  through  Wave  Relay  radios.  The  first 
step  of  integrating  the  QualNet  interface  was  successfully  resolved  after  following 
already  proposed  solutions  from  previous  theses,  as  already  discussed.  The  problems  that 
came  up  during  the  replay  phase,  though,  were  not  solved  within  the  time  available  for 
this  thesis.  The  results  obtained  from  the  overall  STK  simulation  are  presented  in  the 
following  subsections. 

1.  STK 

The  STK  simulation  was  intended  to  give  us  an  insight  into  what  ranges  we 

should  expect  to  establish  communications  between  the  several  nodes  that  were  imported 

in  the  scenario.  The  scenario  involved  two  surface  units,  an  unmanned  aerial  vehicle 

(UAV),  one  ground  vehicle,  and  three  ground  stations.  In  Figure  65,  an  instance  of  the 

simulation  is  shown.  The  white  lines  represent  the  routes  followed  by  the  platforms,  and 

the  brown  lines  represent  the  communication  links  between  the  nodes.  In  this  instance, 

the  range  between  the  two  surface  units  was  the  maximum  observed  in  the  scenario.  The 

height  of  the  antennas  was  set  to  10  meters,  and  the  measured  distance  was  14.6  km.  The 

rest  of  the  ranges  that  were  measured  cannot  be  considered  as  a  basis  since  they  were 

dynamically  formed.  The  altitude  and  route  of  the  UAV,  the  height  of  the  ground 
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stations,  and  the  geography  were  the  parameters  that  affected  the  various  ranges  that  links 
were  closing  between  the  nodes. 


Figure  65.  Wireless  Mesh  Links  of  the  Participating  Nodes 


An  instance  of  the  wireless  mesh  network  simulation  is  depicted  in  this  image,  as 
extracted  from  the  STK  scenario.  The  various  platform-nodes  are  moving  along  the 
routes  shown  in  white.  The  dynamic  forming  of  communication  links  is  represented  by 
the  brown  lines. 

Additionally,  results  were  also  derived  concerning  the  availability  of  the  imaging 
satellites.  A  theoretical  ficld-of-regard  (FOR)  within  which  the  look  angle  could  be 
tasked  is  depicted  by  the  brown  sector  in  Figure  66.  The  time  window  that  the  field  of 
view  (FOV),  depicted  with  a  red  sector,  could  be  tasked  inside  the  FOR  and  the  area  of 
interest  (AOI)  was  calculated  to  nine  minutes  over  a  24-hour  period  from  the  STK  tools. 
Small  intervals  in  which  imaging  sensors  were  not  available  were  reported  as  expected 
and  can  be  easily  assumed  from  Figure  66.  The  scenario,  though,  did  not  have  as  a  goal  to 
demonstrate  continuous  access  but  rather  to  demonstrate  the  availability  of  imaging 
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capabilities  for  low  earth  orbit  (LEO)  cube-satellites.  The  orbits  could  be  redesigned 
accordingly  to  give  the  desired  availability  of  the  sensors. 


Figure  66.  Image  of  Cube-Satellites  in  STK  Scenario 


FOR  and  FOV  of  the  imaging  satellites  are  depicted  in  this  figure  as  extracted  from  the 
3D  graphics  window  of  the  STK  simulation. 


The  total  time  that  communication  links  could  be  established  between  the  imaging 
satellites  and  the  ground  stations  was  calculated  to  22  minutes  from  STK  tools.  In  Figure 
67,  an  instance  from  2D  graphics  of  STK  simulation  is  presented.  The  brown  lines  in  the 
figure  represent  the  closed  links  between  the  ground  stations  and  an  imaging  satellite. 
Part  of  the  crosslink  between  the  satellites  is  also  shown.  The  crosslinks  make  it  possible 
for  the  time  window  of  communications  within  each  orbital  period  to  be  continuous. 
While  these  communication  links  are  available,  they  contribute  to  communications 
between  the  nodes,  especially  when  links  between  ground  stations  are  broken  due  to 
range  and  geographical  restrictions. 
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Figure  67.  Image  of  Cub-Satellite  Links 


In  this  instance  of  2D  graphics  extracted  from  the  STK  scenario,  communication  links 
between  ground  stations  and  satellites  as  well  as  crosslinks  are  depicted  in  brown. 


After  adding  a  constellation  of  communication  relay  satellites,  crosslinks  were 
multiplied.  The  time  window  that  communications  with  the  imaging  satellites  and  thus 
tasking  them  was  possible  was  then  increased  to  three  hours  over  the  24-hour  period  of 
the  simulation  run.  In  this  case,  communication  intervals  were  evident  and  the  tasking, 
even  if  possible  at  the  same  time,  was  not  restricted  in  the  specified  AOI  but  instead,  in 
the  random  locations  of  the  imaging  satellites.  In  Figure  68,  the  communication  relay 
satellites  and  their  orbits  are  depicted  in  white  while  the  imaging  satellites  are  depicted  in 
brown.  The  purpose  for  this  implementation  was,  again,  to  demonstrate  the  ability  to 
increase  the  available  time  for  communicating  with  the  imaging  satellites.  Certain 
limitations  continue  to  exist  since  the  availability  cannot  be  continuous.  However,  the 
benefits  are  obvious  for  the  mesh  networks  of  the  sensor  since  the  contribution  of  the 
satellite  constellation  was  increased  from  22  minutes  to  3  hours. 
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Figure  68.  Communications  Relay  and  Imaging  Cube  Satellite  Constellations 


In  this  instance  of  3D  graphics  extracted  from  the  STK  scenario,  the  two  satellite 
constellations  are  depicted.  The  communications  relay  satellites  and  their  orbits  are 
shown  in  white  and  the  imaging  ones  in  brown. 


2.  QualNet 

The  QualNet  add-in,  although  successfully  integrated  and  configured  for  the 
needs  of  this  scenario,  always  crashed  the  STK  simulation.  Several  attempts  were  made 
to  recover  it  and  run  the  replay  of  QualNet  to  evaluate  the  Wave  Relay  performance 
under  the  constraints  of  the  available  applications.  Unfortunately,  there  was  no  success 
and,  thus,  this  task  is  left  for  future  work  in  this  research  area.  What  we  expect  from  this 
interface  is  that  it  can  give  us  a  more  realistic  range  than  applications  over  Wave  Relay 
could  in  the  field. 

B.  CENETIX  FIELD  EXPERIMENTS 

From  the  several  CENETIX  experiments  that  were  carried  out  in  San  Francisco 
during  the  most  recent  experiments,  the  extracted  results  of  some  of  them  prove  the 
feasibility  of  this  thesis  concept.  Previous  fourth  generation  Wave  Relay  performance, 

integration  of  the  CodeMettle  network-managing  and  monitoring  tool,  as  well  as 
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incorporation  of  divers  and  unmanned  ground  vehicles  (UGVs)  into  the  tactical-network- 
topology  (TNT)  reachback,  demonstrate  the  potential  of  a  system  with  these  capabilities. 
More  detailed  results  for  each  of  these  experiments  are  presented  in  the  following 
subsections. 

1.  Wave  Relay  Performance 

Wave  Relay  was  widely  used  to  network  marine  units  of  the  TNT  testbed.  The 
interest  of  this  thesis  was  to  extract  a  maximum  distance  at  which  node  to  node 
communications  link  could  be  closed.  As  shown  in  Figure  69,  the  maximum  detected 
range  during  this  scenario  was  4.02  km.  This  range  matches  what  was  expected  based  on 
previous  experience  and  CENETIX  field  experimentation.  The  maximum  range  derived 
from  fourth  generation  Wave  Relay  radios  is  obviously  shorter  since  power  is  one-third 
of  what  the  fifth  generation  of  radios  offers. 
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Figure  69.  Wave -Relay  Node  Tracks  Replayed  in  Google  Earth 


Tracks  of  marine  boats  as  Wave-Relay  nodes  as  replayed  from  SA  Replay  Control  page 
on  CENETIX  site  and  displayed  in  Google  Earth.  Source:  “SA  Replay  Control,” 
http://cenetix.nps.edu/SAl/SARep  lay/# 


2.  CodeMettle 

After  the  successful  integration  of  the  CodeMettle  hybrid  network  orchestration 
tool  for  the  monitoring  and  management  of  the  CENETIX  backbone  and  testbed 
reachback,  the  visualization  of  the  current  status  of  the  nodes  and  the  overall  performance 
metrics  of  the  network  were  possible.  As  shown  in  Figure  70,  geographic  information  of 
the  nodes  and  the  overall  network  was  possible  as  well  as  the  capturing  of  their  tracking 
data.  Additionally,  network  traffic  and  latencies  were  visualized,  which  made  it  possible 
to  estimate  the  source  of  any  detected  network  downgrades. 
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Figure  70.  CENETIX  Backbone  Network  Details  via  CodeMettle  Dashboard 


In  this  extracted  instance  of  the  CodeMettle  dashboard,  several  metrics  and  visualizations 
of  the  CENETIX  backbone  and  testbed  reachback  are  presented. 


During  the  monitoring  of  the  wireless  mesh  network  formed  by  the  Wave  Relay 
nodes,  CodeMettle  made  it  possible  to  feed  the  network  operations  center  (NOC)  with 
live  geo-data  that  was  additionally  captured  for  replay  and  analysis  purposes.  As  it  is 
shown  in  Figure  71,  traffic  analysis  by  node  of  the  mesh  network  was  possible  as  well  as 
access  to  further  details  of  the  nodes  such  as  IP  addresses  of  the  nodes,  connectivity 
strength,  and  quality  between  the  nodes.  The  availability  of  these  visualizations  and 
metrics  enhanced  the  situational  awareness  (SA)  of  the  NOC  personnel  and  assisted  in 
decision  making  processes  in  terms  of  corrective  actions  for  a  more  efficient  use  of  the 
network  and  its  nodes’  capabilities.  In  this  thesis  concept,  the  available  information  from 
CodeMettle  could  support  the  efficient  use  of  the  available  bandwidth  based  on  the 
operational  needs.  The  capability  that  CodeMettle  demonstrated  to  perform  hybrid- 
network  orchestration — since  it  is  designed  to  be  network  agnostic — makes  it  a  necessary 
tool  for  future  implementation  of  distributed  operations  solutions. 
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Figure  7 1 .  CENETIX  Wave -Relay  Mesh  Network  Details  via  CodeMettle 
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3.  Wireless  Communication  with  Divers  and  UGV 

Both  UGV  and  divers  were  successfully  incorporated  into  the  CENETIX 
reachback.  In  Figure  72,  the  components  that  made  possible  imagery  data  dissemination 
to  divers  are  demonstrated.  More  analytically,  in  Figure  72  (a),  the  diver  is  shown  ready 
to  dive  after  configuring  his  android  device,  which  was  encapsulated  in  a  waterproof 
case.  Because  of  the  pressure  when  underwater  and,  therefore,  the  inability  to  activate  the 
camera  through  the  touch  screen,  the  solution  of  a  near-field-communication  (NFC)  tag 
was  successfully  implemented.  In  Figure  72  (b),  a  message  is  shown  on  the  screen  that 
informs  the  diver  that  the  captured  image  is  ready  to  be  uploaded  in  the  application  that 
enables  communication  with  the  NOC.  The  image  is  captured  by  activating  the  camera 
through  an  NFC  tag.  In  Figure  72  (c),  an  instance  of  the  application  is  shown,  whereby 
the  diver  receives  in  his  device  an  image  captured  from  an  unmanned  underwater  vehicle 
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(UUV)  in  Keyport  (Seatttle)  through  the  CENETIX  testbed.  In  Figure  72  (d),  the 
equipment  to  connect  the  diver  to  the  CENETIX  testbed  is  shown.  The  buoy  was 
composed  of  a  computer  connected  to  the  CENETIX  testbed  through  the  4G  network  and 
was  connected  through  ethemet  cable  to  an  android  device.  Bluetooth  was  used  for  the 
communication  between  the  tethered  device  and  the  one  that  was  available  to  the  diver. 
In  Figure  72  (e),  an  instance  of  the  application  running  in  the  NOC  is  presented.  Through 
this  application,  dissemination  of  messages  and  images  with  the  diver  was  possible. 


Figure  72.  Enabling  Communications  with  Divers 
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In  this  figure,  components  of  the  implementation  of  incorporating  divers  in  the 
CENETIX  reachback  are  presented.  In  (a),  the  diver  is  ready  to  dive  after  configuring  his 
android  device.  In  (b),  a  message  is  presented  that  the  image  the  diver  captured  by 
activating  the  camera  through  a  near  field  communication  (NFC)  tag,  and  sent  to  the 
NOC,  is  ready  to  be  uploaded.  In  (c),  the  diver  receives  in  his  device  an  image  captured 
from  an  unmanned  underwater  vehicle  (UUV)  in  Keyport  (Seatttle)  through  the 
CENETIX  testbed.  In  (d),  the  equipment  to  connect  the  diver  to  the  CENETIX  testbed  is 
shown.  The  buoy  was  comprised  by  a  computer  connected  to  the  CENETIX  testbed 
through  4G  network  and  was  connected  through  ethemet  cable  with  an  android  device. 
Bluetooth  was  used  for  the  communication  between  the  tethered  device  and  the  one  that 
was  available  to  the  diver.  In  (e),  an  instance  of  the  application  running  in  the  NOC  is 
presented.  Source:  “TNT-MIO  Experimentation:  Video  and  snapshots  archive”  , 
http://cenetix.nps.edu/cenetix/tnt.asp  and  author’s  captured  images  in  the  field. 
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Finally,  the  UGV  was  also  successfully  networked  to  the  CENETIX  testbed 
through  Wave  Relay.  A  ground  station  acting  as  a  gateway  interconnected  the  Wave 
Relay  mesh  network  with  the  TNT  testbed  via  satellite  communications. 


Figure  73.  Enabling  Communications  with  a  UGV 


In  (a),  the  UGV  connected  through  Wave  Relay  to  the  TNT  testbed  is  shown.  In  (b),  the 
Wave  Relay  radio  and  the  application  that  the  UGV  runs  for  the  detection  of  radioactive 
materials  is  presented.  Source:  “TNT-MIO  Experimentation:  Video  and  snapshots 
archive”  ,  http://cenetix.nps.edu/cenetix/tnt.asp. 


C.  IMAGE  FUSION  AND  CLASSIFICATION 

The  methodology  that  was  followed  for  this  part  of  the  experiments  was  described 
step-by-step  in  the  previous  chapter.  The  datasets  presented  in  Table  9  were  processed 
twice  through  the  classification  procedure  using  a  MATLAB  script.  The  first  time,  the 
test  set  was  kept  the  same  for  1000  trials.  The  second  time,  10  ANNs  were 
simultaneously  trained  with  10  random  test  sets,  respectively.  In  an  effort  to  yield 
unbiased  results  in  the  first  case,  the  tests  comprised  the  same  images  for  the  groups  of 
datasets  that  had  the  same  number  of  images  as  in  the  test  set.  The  final  classification 
performance  results  for  each  dataset  is  our  final  aim  to  evaluate  whether  multi-sensor 
image  fusion  adds  value  in  the  classification  process. 
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Table  9.  Datasets  for  Testing 


Trial 

No 

Description 

No  of 
images 

No  of  test 
images 

Visual  images  from 

1  DSC640 

62 

7 

Thermal  images  from 

2  DSC640  62  7 

MSX  images  from 

3  DSC640 

62 

7 

Fused  features  from 

4  visual  and  thermal  62  7 

images  from  DSC640 

^  Mixed  visual  images 

from  both  cameras 

85 

9 

_  Mixed  MSX  images  __  _ 

b  r  .  .  ob  y 

from  both  cameras 

7 

MSX,  thermal,  and 
visual  images  from 
both  cameras 

232 

23 

1.  Feature  Extraction  and  Training  Results 

The  MATLAB  script  used  to  obtain  the  results  is  graphically  described  in  Figure 
74.  After  the  image  processing  steps,  each  image  set  was  scanned,  so  SURF  features 
could  be  extracted.  Consequently,  the  ten  strongest  features  were  turned  into  vectors  and 
assigned  a  binary  class  of  01  or  10,  matching  to  the  classes  of  powerboats  and  sailing 
boats,  respectively.  The  descriptors  had  640  elements  for  each  image  in  all  datasets, 
except  for  dataset  four  in  Table  9  in  which  the  fused  features  resulted  in  descriptors  with 
a  doubled  size  of  1,280  elements.  The  ANN  was  then  trained,  and  the  extracted  weights 
were  stored.  The  test  set  was  finally  gone  through  the  stored  ANN  weights  and  the 
classification  performance,  as  well  as  the  cross-entropy  and  the  MSE  were  stored  for  each 
trial  to  be  used  as  metrics  for  the  evaluation  of  the  overall  results. 
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Figure  74.  Image  Classification  Steps 
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After  comparing  the  extracted  features  for  each  dataset,  it  was  observed  that 
SURF  features  were  indeed  repeatable  but  different  for  the  distinct  datasets  of  the  images. 
The  depiction  of  this  result  is  presented  in  Figure  75. 
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Figure  75.  Ten  Strongest  SURF  Features  for  the  Different  Image  Sets 


(c) 

The  ten  strongest  SURF  features  were  always  repeatable  but  different  for  visual  in  (a), 
thermal  in  (b),  and  multi-spectral  in  (c).  All  the  images  were  in  grayscale. 


As  discussed  in  Chapter  III,  early  stopping  was  applied  during  all  the  training 
sessions  as  the  results  were  better  than  the  ones  obtained  when  it  was  not  applied,  even 
though  cross-entropy  was  smaller  in  the  latter  cases.  This  fact  was  evidence  that  the 
implementation  of  early  stopping  succeeded  in  avoiding  overfitting. 

Furthennore,  classification  performance  during  training  was  constantly  monitored 
and,  in  the  case  of  the  multispectral  dataset,  demonstrated  very  good  results.  The 
confusion  matrix  and  the  ROC  curves  from  one  training  session  are  presented  in  Figure 
76.  From  the  monitoring  of  the  training  processes,  it  was  concluded  that  the  classification 
of  the  sailing  boats  class  demonstrated  constantly  higher  percentages.  The  factors  that 
possibly  contributed  to  that  result  were  the  larger  number  of  sailing  boats  in  the  dataset 
compared  to  that  of  powerboats,  as  well  as  the  fact  that  powerboats  had  a  wide  variety  of 
shapes  and  structures. 


96 


Figure  76.  Confusion  Matrix  and  ROC  Curves  for  Training  and  Validation 


Training  Confusion  Matrix  Validation  Confusion  Matrix 


Target  Class 


Target  Class 


The  presented  confusion  matrices  and  curves  refer  to  one  training  session  of  the  1000 
trials  over  the  10  NNs. 


2.  Testing  Classification  Results 

After  the  training  process  was  completed  and  the  parameters  and  weights  of  the 
NN  were  stored,  the  test  set  was  fed  into  the  NN.  The  results  were  also  monitored  and 
stored.  Fourteen  sets  of  plots  were  derived  from  the  MATLAB  code  implementation 
concerning  the  metrics  and  results  relevant  to  the  classification  performance  of  the  test 
sets.  The  set  that  represents  the  best  results  are  presented  and  described  to  support  the 
overall  analysis.  This  set  was  obtained  from  the  MSX  images  from  dataset  3  of  Table  9 
and  for  the  case  of  the  ten  different  ANNs.  Figure  77  shows  the  results  obtained  from  an 
individual  trial. 
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Figure  77.  Confusion  Matrix  and  ROC  Curve  for  One  Trial 


Confusion  Matrix  for  Trial  No:  100  of  10NN 


Target  Class 


The  presented  confusion  matrix  and  ROC  curves  refer  to  one  testing  session  of  the  1000 
trials  over  the  10  NNs. 

The  stored  values  of  each  testing  process  were  plotted  to  present  an  overall 
pattern  of  the  classification  performance.  In  Figures  78  and  79,  the  average  classification 
over  100  trials  for  each  of  the  ten  NNs  and  the  individual  1,000  trials  are  presented.  The 
average  classification  performance  of  this  set  was  above  62  percent,  but  the  individual 
results  per  trial  demonstrated  a  wide  spread  from  0  to  100  percent.  The  previous 
observation  denotes  that  there  were  some  cases  in  which  the  NNs  did  not  generalize  well 
on  the  test  set.  In  the  majority  of  the  cases,  the  powerboat  class  yielded  the  worst 
classification  rates.  Possible  reasons  for  this  result  could  be  the  smaller  number  of  sample 
images  compared  to  the  sailing  boats  as  well  as  their  variety  in  shape  and  type. 
Conversely,  sailing  boats  had  less  varying  shapes. 
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Figure  78.  Average  Correct  Classification  Performance  for  the  10  NNs 
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Figure  79.  Correct  Classification  Performance  for  Each  of  the  1000  Trials 

of  the  10  NNs 
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Additionally,  cross-entropy  and  mean-squared-error  (MSE)  were  plotted  for  each 
trial  and  used  as  a  measure  of  fit.  The  lower  values  represent  NNs  that  better  generalized 
to  the  test  dataset.  In  Figures  80  and  81,  the  obtained  results  for  these  metrics  over  1000 
trials  are  shown. 
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Figure  80.  MSE  over  1000  trials 
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Figure  81.  Cross-entropy  over  1000  Trials 
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As  previously  discussed,  the  smaller  the  derived  values  of  cross-entropy  and 
MSE,  the  better  the  generalization  over  the  test  value  as  well  as  the  classification  rates. 
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3.  Summary  of  Image  Classification  Results 

The  classification  results  for  all  the  datasets  and  the  case  of  10  NNs  are  presented 
in  Table  10.  The  best  rates  were  obtained  for  the  MSX  images  closely  followed  by  the 
thermal-image  sets.  In  general,  when  images  only  from  the  better  camera  were  used,  the 
obtained  results  were  clearly  better.  Additionally,  low-level  fusion  of  the  MSX  images 
clearly  gave  better  classification  results  than  medium-level  fusion  of  the  SURF  features 
obtained  from  the  visual  and  thennal  images. 


Table  10.  Overall  Classification  Results 


Trial 

Description 

No  of 

No  of  test 

Test  set 

Average 

Average 

Average 

No 

images 

images 

Classification 

MSE 

Cross-entropy 

(%) 

i 

Visual  images  from 
DSC640 

62 

7 

10  random 

58.8857 

0.3193 

0.13748 

2 

Thermal  images  from 
DSC640 

62 

7 

10  random 

60.3714 

0.31225 

0.13277 

3 

MSX  images  from 
DSC640 

62 

7 

10  random 

62.3714 

0.3012 

0.12501 

Fused  features  from 

4 

visual  and  thermal 
images  from  DSC640 

62 

7 

10  random 

55.2 

0.3424 

0.15248 

5 

Mixed  visual  images 
from  both  cameras 

85 

9 

10  random 

50.6 

0.35521 

0.1632 

6 

Mixed  MSX  images 
from  both  cameras 

85 

9 

10  random 

54.4778 

0.33654 

0.15019 

MSX,  thermal,  and 

7 

visual  images  from 

232 

23 

10  random 

54.5522 

0.33704 

0.15055 

both  cameras 

Derived  results  after  averaging  100  trials  for  10  ANNs  trained  with  10  random  test  sets. 
While  the  test  sets  were  the  same  in  each  one  of  the  100  trials,  training  and  validation  sets 
were  randomly  selected  with  a  ratio  of  7:3,  respectively.  The  best  classification  result  was 
obtained  from  the  MSX  image  set  captured  with  the  FLIR  DSC  640  camera,  which  had 
the  best  resolution. 
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V.  CONCLUSIONS 


This  thesis  demonstrated  simulation  modeling  of  the  concept  of  automatic  target 
recognition  (ATR)  via  mesh  networking  of  imaging  sensors  as  part  of  the  concept  of 
distributed  management  of  littoral  operations.  Proof  of  the  feasibility  for  specific  parts  of 
the  overall  concept  was  verified  with  simulations  and  field  experimentation.  The 
parameters  for  the  simulations  were  based  on  commercial  and  already  tested  solutions 
either  in  field  experiments  by  CENETIX  field  experimentation  or  in  published  results. 

Initially,  different  platfonns  with  imaging  capabilities  were  simulated  as  nodes  in 
an  ad-hoc  mesh  tactical  network  scheme.  The  quality  of  service  (QoS)  of  this  network 
was  then  studied.  The  extracted  results  from  STK,  field  experiments,  and  Qualnet  showed 
the  feasibility  of  using  this  type  of  network  for  the  timely  dissemination  of  critical 
imagery  data.  Furthermore,  images  of  boats  were  captured  in  the  field  to  simulate  images 
taken  from  sensors  of  the  networked  platforms,  which  cover  the  visible  and  part  of  the  IR 
spectra.  Captured  images  were  selected  to  have  noisy  background  that  is  usually  met  in 
littoral  environments.  Low-  and  medium-level  fusion  were  applied  on  three  types  of 
imagery  data — visual,  thermal,  multi-spectral  (MSX) — and  speeded-up  robust  features 
(SURF)  were  extracted  from  them.  Results  have  proven  that  MSX  images  from  low-level 
fusion  contribute  to  better  classification  results  in  this  thesis’  binary  classification 
problem.  Artificial  neural  networks  (ANNs)  were  used  to  derive  the  classification  results 
and  demonstrated  the  ability  to  obtain  these  results  in  a  timely  manner  that  can 
accommodate  near  real-time  classification  of  COIs  at  the  tactical  level. 

Finally,  CENETIX ’s  latest  experimentation  results  were  used  to  demonstrate  not 
only  the  operational  advantages  of  mesh  networks  but  also  the  capabilities  of  managing 
them.  Hence,  this  thesis  suggests  that  major  units  should  have  the  capability  to  deploy 
mesh  tactical  networks  augmented  by  sensor  management,  operational  databases, 
geographic  infonnation,  and  an  appropriate  level  of  automation  for  target  recognition, 
especially  when  assuming  the  role  of  mother  ships  for  other  platforms.  More  specific 
conclusions  from  the  research  in  mesh  tactical  networking  and  image  fusion  and 
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classification,  as  well  as  future  work  recommendations  are  discussed  in  the  following 
sections  of  this  chapter. 


A.  MESH  TACTICAL  NETWORKING 

The  scenario  implemented  in  Systems  Tool  Kit  (STK)  demonstrated  great 
potential  to  evaluate  equipment  capabilities  and  test  them  in  operational  scenarios  with 
simulated  platfonns.  The  simulated  mesh  tactical  radios  demonstrated  increased  ranges 
from  already  tested,  in-the-field,  fourth-generation  Wave  Relay  radios.  This  result  seems 
reasonable  since  power  was  three  times  more  than  the  previous  version  and  technologies, 
such  as  multiple-input  multiple-output  (MIMO),  had  embedded. 

The  QualNet  add-in  was  successfully  installed,  but  during  the  attempts  to  run  its 
replay  and  test  the  perfonnance  of  the  network  with  a  simulated  application,  the  STK 
scenario  crashed.  What  would  be  expected  had  the  scenario  run  successfully,  would  be 
that  the  ranges  the  actual  applications  would  demonstrate  acceptable  performance  would 
be  less  than  the  ones  that  the  communications  link  were  closing.  Furthermore,  the  various 
ranges  that  the  fifth  generation  of  Wave  Relay  gave  as  results  for  establishing 
communication  between  the  assets  used  in  the  simulation  showed  that  these  ranges  are 
highly  dependent  on  the  geography  and  the  routes  of  the  assets. 

Additionally,  field  experimentation  demonstrated  very  promising  results  both  for 
communications  with  divers  and  the  UGV  that  could  become  part  of  the  mesh  network. 
Gateways  connecting  mesh  network  with  4G  and  satellite  links  were  used  to  establish 
communication  between  the  remote  platforms  and  the  network  operations  center  (NOC) 
through  the  CENETIX  TNT-MIO  testbed.  In  all  the  cases,  imagery  data  dissemination 
was  successful.  Especially  in  the  case  of  the  diver,  obtained  results  revealed  new  ways 
that  could  be  developed  to  allow  for  continuous  communication  with  special  forces  even 
when  underwater. 

Finally,  CodeMettle  was  successfully  implemented  for  management  and 
monitoring  of  the  wireless  mesh  tactical  network.  Results  showed  that  real  time 
monitoring  of  the  performance  of  the  network  and  the  nodes  through  the  dashboard 
supported  decision-making.  The  reasons  that  the  network  demonstrated  downgrades  and 
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reduced  performance  could  be  easily  recognized.  Corrective  actions  could  then  be 
detennined.  An  overall  implementation  would  eventually  enhance  situational  awareness 
(SA)  by  taking  into  consideration  the  network  parameters  for  a  more  effective 
deployment  of  the  available  assets. 

B.  IMAGE  FUSION  AND  CLASSIFICATION 

Operational  experience  and  previous  experiments  have  proven  that  operators  of 
imaging  sensors  find  beneficial  the  use  and  projection  of  multi-spectral  imagery  data 
from  low-level  fusion  as  they  reveal  richer  combined  information.  The  fusion  of  thennal 
and  visual  images  in  our  experiment  obviously  enhanced  the  perception  of  the  operator 
by  revealing  details  that  allowed  for  faster  identification  of  specific  spots  or  objects  in  the 
images.  Therefore,  current  capability  of  imaging  sensors  to  shift  from  a  visual  to  IR  mode 
is  not  as  efficient  as  adding  a  multispectral  mode.  This  thesis  has  proven  that  the  same 
conclusion  holds  for  the  case  in  which  computer-vision  methods  are  implemented  in 
automatic  target  recognition  (ATR).  Multispectral  images  gave  higher  classification 
results  than  all  the  other  sets  of  images  that  were  tested,  even  when  classification  rates 
were  not  sufficiently  high.  However,  since  several  limitations  were  known  from  the 
beginning  of  this  research  concerning  the  equipment  used  as  well  as  the  challenges  the 
noisy  background  poses,  the  low  classification  rates  were  expected,  and  the  results  seem 
to  be  promising. 

In  contrast,  when  all  the  types  of  images  were  fused  in  several  ways  or  medium- 
level  fusion  of  extracted  features  from  them  was  applied,  the  classification  perfonnance 
did  not  yield  better  results.  The  same  result  occurred  even  when  multi-spectral  images  of 
the  two  cameras  were  tested  as  one  set,  as  shown  in  Table  10.  Therefore,  we  can  drive  the 
conclusion  that  adding  low  resolution  images  results  in  worse  classification  performance, 
even  for  some  sets  in  which  the  number  of  images  was  four  times  larger  than  the  set  that 
yielded  the  best  classification  perfonnance. 

Another  conclusion  derived  from  the  suggested  scheme  for  ATR  is  the  capability 
to  exploit  the  received  imagery  data  in  a  timely  manner.  The  combination  of  SURF 
features  with  NNs  proved  to  be  extremely  fast  and  robust.  Every  time  SURF  features 


105 


were  extracted  from  the  image  sets,  results  concerning  the  interest  points  were  always  the 
same,  proving  their  repeatability.  The  subsequent  use  of  NNs  did  not  give  the  same 
results  as  initial  weights  were  not  the  same  at  each  training  session.  For  the  testing  phase, 
though,  and  after  the  weights  from  training  the  NN  were  stored,  the  use  of  the  same  test 
set  yielded  exactly  the  same  results  in  all  the  trials. 

Finally,  we  can  conclude  that  the  combination  of  descriptors  based  mainly  on 
SURF  features  and  NNs  could  set  the  frame  for  a  development  of  an  algorithm  that  yields 
high  classification  rates  in  ATR.  Sailing  boats  demonstrated  almost  perfect  classification 
rates  in  the  majority  of  the  trials,  and  the  main  reason  behind  the  fairly  low  classification 
rates  is  considered  to  be  the  smaller  number  and  variety  of  types  of  the  powerboats. 
Additionally,  the  reduction  of  the  number  of  SURF  features  to  ten,  even  though  it 
removed  most  of  the  outliers  generated  from  the  background,  did  not  work  as  well  in  all 
the  images. 

C.  RECOMMENDATIONS  FOR  FUTURE  WORK 

The  broad  topic  of  this  thesis  reveals  many  challenges  for  future  work  and  further 
research  in  the  overall  concept  but  also  in  the  specific  topics  that  were  more  thoroughly 
examined.  Specific  recommendations  for  mesh  tactical  networking  and  multi-sensor  data 
fusion  and  classification  are  discussed  in  the  following  subsections. 

1.  Multi-Sensor  Data  Fusion  and  Classification 

The  use  of  only  one  type  of  feature  in  our  binary  problem  yielded  acceptable 
classification  rates,  especially  for  the  sailing  boat  class.  The  use  of  more  sophisticated 
imaging  sensors  and  the  association  with  metadata  from  environmental,  motion,  and 
relevant-position  conditions  would  filter  out  not  well-matching  images  for  the  training 
process  and  would  provide  more  features.  Feeding  the  NN  additional  features — such  as 
size  or  shape  characteristics  extracted  from  the  available  metadata — could  further 
improve  the  classification  rates. 

Further  research  could  also  be  carried  out  to  determine  the  types  of  imagery  or 
other  data  coming  from  multiple  dissimilar  sensors  that  could  be  fused  to  provide  richer 
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and  more  robust  features  under  specific  conditions  such  as  rain  or  darkness.  Synthetic 
aperture  radars  (SARs)  or  IR  sensors  in  MWIR  and  LWIR  could  be  some  to  consider. 

Additionally,  since  SURF  features  are  very  robust,  the  reverse  problem  could  be 
implemented  in  potential  ATR  solutions.  Filtering  out  images  of  a  specific  or  only  a 
small  group  of  COIs,  subsequently  training  an  ANN,  and  performing  only  a  binary 
classification  could  yield  better  classification  results.  This  process  could  even  be 
completed  at  the  platform  detecting  the  COIs,  so  the  feedback  to  the  network  could 
provide  matching  data.  This  type  of  solution  could  potentially  help  intelligence, 
surveillance,  and  reconnaissance  (ISR)  operations  through  the  use  of  swarms  of  platforms 
with  imaging  sensors.  Monitoring  of  multiple  screens  would  then  be  impossible,  and 
feedback  that  would  only  highlight  targets  of  interest  could  enhance  SA  and  reduce  an 
operator’s  work  load. 

The  training  and  testing  processes  could  also  be  improved  by  researching  and 
testing  different  techniques  that  could  enhance  NN  performance.  Applying  deep  NN 
techniques  or  adding  more  layers  in  the  ANN  could  result  in  better  generalization  and, 
consequently,  better  classification  performance.  Finally,  background  extraction  or 
segmentation  of  the  image  and  application  of  high-level  fusion  could  potentially  improve 
extracted  results. 

2.  Mesh  Tactical  Networking 

Even  if  STK  gave  results  that  could  be  used  in  simulating  scenarios  with  mesh 
tactical  networks  through  Wave  Relay  radios,  further  analysis  should  be  attempted. 
QualNet  analysis  would  eventually  give  reduced  and  different  ranges  for  video  streaming, 
data,  and  voice  applications.  CodeMettle  could  then  be  used  for  the  evaluation  of  the 
results  in  the  field.  Finalizing  the  simulation  parameters  in  STK  would  then  be  feasible, 
and  extended  simulation  scenarios  could  be  tested.  This  process  would  be  highly 
beneficial  for  the  evaluation  of  future  field  experiments  and  operational  scenarios  in 
different  geographic  regions.  Difficulties  and  potential  problems  could  then  be  detected, 
and  the  final  plan  could  be  executed  after  the  appropriate  corrections. 
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Finally,  the  implementation  of  the  proposed  concept  of  distributed  management  of 
littoral  operations  in  the  tactical  level  could  be  further  tested  in  the  field.  Considering  the 
overall  results  of  this  research  it  seems  that  the  use  of  mesh  tactical  networks  would  make 
possible  the  availability  of  ISR  capabilities — UAVs  and  Cubesats — at  the  tactical  level 
and,  thus,  the  exploitation  of  available  information  in  a  timely  manner.  One  of  the 
benefits  could  be  to  enhance  the  capabilities  of  units  without  the  need  of  modernizing 
them  by  adding  an  appropriate  kit  and  networking  appropriate  platforms  according  to  the 
missions.  This  kit  could  be  easily  removed  and  attached  to  other  platforms  when 
necessary.  Solutions  could  then  be  found  in  communications  and  ISR  limitations  in 
littoral  operations  to  enhance  SA  and  timely  and  independent  exploitation  of  infonnation. 
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APPENDIX.  IMAGE  TO  IMAGE  REGISTRATION  WITH  ENVI 

1 .  Open  ENVI  Classic 


Figure  82.  ENVI  Classic  Menu  Bar 
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File  Basic  Tools 

Classification  Transform  Filter  Spectral  Map  Vector  Topographic 

Radar  Window  Help 

2.  Open  the  images  to  be  registered  by  selecting  File  — »Open  Image  File 
from  the  ENVI  Classic  menu  bar. 

3.  Load  the  RGB  images  in  different  displays 


Figure  83.  Images  Shown  in  Different  Displays 
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4.  Select  Map  —*■  Registration  — >  Select  GCPs:  Image  to  Image  from  the 

ENVI  Classic  menu  bar  and  define  the  displays  for  the  Base  and  the  Warp 
image  that  pops  up. 


Figure  84.  Define  Displays 
&  Image  to  Image  Registra... 

Select  displays  containing  images: 


Base  Image  Waip  Image 

Display  #1 

Display  #2 

1  Display  U2 

Selected  Item:  Selected  Item: 

Display  #1 

Display  #2 

OK 

Cancel 

5.  The  Ground  Control  Point  (GCP)  selection  pops  up.  After  the  selection  of 
the  corresponding  points  in  both  images,  on  the  GCP  selection  window 
select  File  — ►  Save  GCPs  to  ASCII  from  the  same  window. 


Figure  85.  Ground  Control  Point  Selection  Window 
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Figure  86.  Corresponding  Points  Selection 
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Figure  87.  Saving  GCPs 


6.  Then  select  Map  — >  Registration  — >  Warp  from  GCPs:  Image  to  Image 

from  the  ENVI  Classic  menu  bar  and  save  the  output  from  the  pop  up 
window  after  selecting  the  Warp  and  the  Base  image 
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Figure  88.  Select  Warp  and  Base  Image 


9 


Input  Warp  Image 


Select  Input  File: 

File  Information: 

image4 

image3 

image3 

image3 

[Memory  7]  (1280x%(bc3) 

[MemoryS]  (128Qx960<3) 

[Memory5]  (64(bc48Qx3) 
image2 

[Memory4]  (640x480x3) 

[Memory3J  (1280x960x3) 
image 

[Memory 2]  (64(k48Qx3) 

[Memory  1]  (128Qx96Cbc3) 

OK  Cancel 

Previous 

Open  ▼ 

Figure  89.  Save  the  File 
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7.  Load  the  Base  original  image  and  the  output  image  on  ENVI  and  try 

moving  the  indicator  in  the  transparency  tool  on  the  menu  bar  until  you 
get  the  result. 


Figure  90.  Registration  in  ENVI 
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